kavezo / zipzap

ZipZap: MagiReco NA Private Server

Please be mindful when sharing. We don’t want Aniplex shutting down NA yet again, so please keep this off of mainstream social media sites. In fact, when you do share, please add a reminder to keep it off social media.

You need to have a computer to run the server on, and a device with MagiReco to connect to it (both emulator and real physical device work).

To update to a new release: download the release, unzip it without overwriting the old one, then copy your data/user folder from the old one to the new one. If you don't want to install a new cert to your phone every time you update, also copy the (windows/nginx_windows/)nginx/conf/cert from the old to the new.

On the computer: installation and running

For Windows, please download a release rather than the source code.

The first time before you run the server on Windows, you need to do some setup:

1. Run setupServer.bat.
2. A cmd window should pop up. Wait until it closes by itself.
3. Check to make sure there is a file named ca.crt in the ZipZap folder.

You only need to do this once. (Although if you ever run setupServer.bat again, you'll have to repeat step 3.)

Then, to actually run the server, run startServer.bat. A cmd window should pop up without immediately closing; don't close it until you want to stop the server. That's all.

Alternate installation method

An alternate method is available that will run ZipZap in its own self-contained Linux virtual machine. This method can be used on Windows, Mac and Linux. For this, you will need to download the source code rather than the release.

You will need to download and install VirtualBox (be sure and also download and install the VirtualBox Guest Additions) as well as Vagrant. Both of these apps are completely free for personal/non-commercial use. Once these are installed, simply open a terminal in the server window and type vagrant up. This will automatically configure ZipZap and run it.

In some rare cases, Vagrant will ask you for which network interface you would like to use as the bridge interface. Usually the topmost choice is the right one, but in some rare instances you will have to choose one of the other options. If the option you choose didn't work, run vagrant destroy followed by vagrant up and try a different choice.

Once the configuration process is complete, the script will display the VM's IP address. On your device or emulator, set the DNS server to that IP address, then open a web browser and go to https://magica-us.com. You should see a page asking you to install the custom SSL certificate. Follow the instructions to do this. Now you should be able to run Magia Record and connect to your private server!

To start up the private server VM, run vagrant up; and to shut it down run vagrant halt.

On your phone/tablet/emulator: connecting to the private server

0. Run the server on your computer, and keep it running.
1. Find the IP address of the computer you're running the server on. I've only tested this with a local IP address (ones that start with 192.168) but external IP addresses should work.
2. Change your DNS settings to point to your IP address. On iOS you can do this by editing the network settings as you would with a proxy (just make sure to edit DNS, not proxy, settings), but on Android emulator I've had to use a different app. The one I use is just called DNS Changer, and its icon is a blue circle with DNS in it.
3. If this is the first time you're playing, go to https://android.magica-us.com. (The https is important.) Download the link labled ca.crt and install it using the information in the links provided.

But my network is really, really, really weird and DNS messes it up!

2. Edit the config.json in the top level folder of the server so that the "useDNS" key is set to false
3. Use a hosts editor on your phone/emulator to point android.magica-us.com or ios.magica-us.com (based on your OS) to the server computer's local IP address.
4. Everything else is normal. If this is the first time you're playing, go to https://android.magica-us.com and install the cert.

Note that this does not redirect other tracking websites. I honestly don't think it should be a huge problem for now, but sooner or later I'll have to stop being lazy and add them in this README oof

A note about Android 7 (Nougat) and above

Due to a change that Google made starting in Android 7 (Nougat) user-installed CA certificates are no longer trusted by apps. In order to play on devices and/or emulators running Android 7 and above, you'll either have to root, or play using the APK linked on https://android.magica-us.com. (The APK only appears in Windows releases, unfortunately.)

---

Currently supported functions:

• displaying any page in the app (api/page.py)
  • as well as displaying anything in the archive
  • and listing memoria
• improving magical girls (api/userCard.py)
  • level up
  • magia level up
  • awaken
  • limit break
  • customize magical girls' looks (in disks, etc.)
• making teams with magical girls and memoria (api/userDeck.py)
• managing memoria (api/userPiece.py and api/userPieceSet.py)
  • level up
  • limit break
  • making memoria sets
  • putting memoria into the vault and taking them out
  • locking memoria
• gacha (api/gacha.py)
  • pull premium, x1 and x10, using stones and tickets
    • this currently includes all limited megucas and memes, and welfare ones as well. no reason not to lol
  • pull normal
  • view history
• changing user settings (api/gameUser.py, api/user.py, api/userChara.py)
  • set name
  • set comment
  • set leader
  • set background (only two backgrounds are available, but...)
• buying things from the shop (api/shop.py)
  • spending items to get items
  • spending gems to get packs
  • you can get megucas too, but they won't show up in your present box and instead you'll just have them
• playing and clearing quests (api/quest.py, api/questEndpoints)
  • viewing the story before each quest
  • spending and recovering AP
  • leveling up user and magical girls at the end of a quest
  • clearing quest missions and getting rewards
  • dropping items -- currently droprate is set to 2 for each item
  • Ashley will support you
• playing mirrors battles (api/quest.py, api/arena.py, api/questEndpoints)
  • play against one of three giant megucas who will kill you
  • earn Mirrors points, level up, and read the Mirrors story as you do
  • earn Mirrors coins for wins and losses
  • spending and recovering AP
• daily missions (api/userDailyChallenge.py)

Currently missing functions:

• can't clear/accept non-daily challenges
• some quests are missing story and enemy data
  • especially labyrinths
• can't continue quests
• real NPCs don't show up

What's next?

I coded very fast, and very not well, because I wanted to get as many features out before the 30th. Code quality is still important to me, though, and I really don't want to get anything done outside of the basics before improving it enough that maintenance will not be horrendous. But before the 30th, my priority is to get all the basic functions implemented so that we won't have to rely on hitting the real server to figure out what it does for some important thing like battles.

The features are in order of the most overlap with the knowledge I have currently, to the least, because when I implement a new feature I don't know much about at least half of the time is spent researching how it fits in with all the user's data.

• implement missions
• implement announcements
• add unit tests
• refactor for online server
  • compress user data, so it's not huge jsons with info we don't need
• add support for events
• add support for multiple users
• add support for finding other users, following and using supports

---

If you have suggestions or want to help, please contact me at u/rhyme_or_rationale or rins_fourier_transform#2303. And feel free to send a pull request anytime~

zipzap's Issues

Still possible to start from scratch?

I am not really sure what the problem is. I had to move to another android ROM and my my back up of the EN App’s data is somehow broken.
I’m getting connection errors but I am honestly not sure what exactly is causing them.
I am not very advanced in coding yet and wanted to start figuring out how it works to maybe eventually be able to fix bugs but now I can’t even get the game to redownload the data (using certificate, DNS change, and apk from this repo) the screen just stays black and I’m apparently getting 443 errors.
Is the supplement server this used now gone or something? And if does somehow know where I can get the missing data to make the app work again? I‘m really devastated at this point. (I have root so I could basically just copy paste the data if I had a complete copy, if necessary.)

Edit: I found that my older backups from more than half a year ago at first would not restore either but after trying around for a bit I finally got it to restore the data. But the strange issue of 443 errors remain so I doubt the issue lies with anything but the local server (it does seem to connect but only return 443 errors). I’ll need to try to investigate what causes this… I’ll leave the issue open for now as I can not for sure say that the data actually works and also still don’t know if a redownload would be possible.

Black screen on bootup

On the latest update, I'm able to do nearly everything fine. I was able to set up the server, install the certificate and profile, and I was able to boot up the server with seemingly no problems (telling me the app and dns servers have started). When I boot up magireco (after changing my DNS settings) it shows the usual logos and age disclaimer and then it very briefly shows the loading screen, then it goes black with nothing booting up at all (no gallery mode, no normal title screen). I can tap the screen but it does nothing. I saw others saying they had this problem on android but I'm on an ipad so I'm not sure if that correlates with the problem. I wish I could give more information but I think this is all I have, if there's anything else I can do to help describe the problem please let me know.
Thank you contributors for your hard work on this project!!

How do i correctly build the server?

I want to play with the latest fixes, but the build_all_for_windows.bat is only building gevent_server and generate_cert. Looking for advice on how to build the server completely.

Still can't use newest version: older version now breaks as well

I previously raised an issue about how ZipZap was crashing any time I attempted to do anything beyond the home screen, so I rolled back to version 0.6 to fix it. Apparently the problem was something to do with labyrinth quests, but it still hasn't been resolved. Now the older version has stopped working, as well, not even loading the server at all, with the error message:

nginx version: nginx/1.19.4
Traceback (most recent call last):
  File "gevent_server.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "d:\0w0\magireco\na_server_flask\zipzap\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
  File "app.py", line 18, in <module>
  File "util\homuUtil.py", line 145, in startCron
  File "util\homuUtil.py", line 101, in resetShop
AttributeError: 'datetime.timedelta' object has no attribute 'replace'
[5944] Failed to execute script gevent_server

This seems to have been fixed in the latest version, but everything else breaks for me with that version. Since posting my last issue, I've picked up a bit of Python knowledge, and was going to just try to fix it myself, but the files seem to be bundled in an exe, and are thus locally inaccessible. If you could point me to the offending file in the repository, perhaps I could try fixing it that way, though I don't know how I would test it if the file can't run individually. Any advice?

Alternatively, if this has something to do with my specific userdata (a guess suggested on the previous issue), where would the conflict be, and how could I fix it locally?

startServer.bat error

Sorry, it's me again!
Getting an error with the latest version-I complete the instructions successfully all up until running startServer.bat. The second window that pops up reads the following out to me and immediately closes:

Traceback (most recent call last):
File "gevent_server.py", line 5, in <module>
File "<frozen_importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "d:\0w0\magireco\na_server_flask\zipzap\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
File "dnserver.py", line 7, in <module>
socket.gaierror: [Errno 11001] getaddrinfo failed
[716] Failed to execute script gevent_server

As far as I can tell the private server is unable to boot up from there. The first window that popped up just reads out:
nginx version: nginx/1.19.4
and nothing more.

Error when transferring over old user data

The newer versions seem to be working fine until I transfer over my user data, then I get an error upon running startServer.bat. I've tested to make sure it works with the default user data and it does so I think it must be something with one of the files in my user data.

nginx version: nginx/1.19.4
Traceback (most recent call last):
  File "gevent_server.py", line 2, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "d:\0w0\magireco\na_server_flask\zipzap\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
  File "app.py", line 13, in 
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "d:\0w0\magireco\na_server_flask\zipzap\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
  File "api\arena.py", line 6, in 
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "d:\0w0\magireco\na_server_flask\zipzap\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 493, in exec_module
  File "util\dataUtil.py", line 47, in <module>
  File "util\dataUtil.py", line 20, in createIndex
  File "util\dataUtil.py", line 23, in createIndexFromList
  File "util\dataUtil.py", line 23, in <dictcomp>
  File "util\dataUtil.py", line 7, in <lambda>
TypeError: 'NoneType' object is not subscriptable
[8780] Failed to execute script gevent_server

.

.

Forced boot into Gallery Mode

As of a little while after NA magireco has shut down, the private server no longer works as the game seems to force itself into gallery mode shortly after starting up the game/before you get to the title screen. There's seemingly no workaround for this on our side; I thought maybe setting the time backwards on my device could trick the game as I had seen people set the time forwards before the shutdown to access gallery mode in advance, but setting the time backwards just gave me a connection error 1 and no bootup at all.

Not an issue, just a question.

Hi! I'm a budding-ish programmer myself (I've been programming as a hobby since high school) and I know how to code, but not how to create servers for dead mobile games. Everywhere I go, people just say if I have to ask questions, then it's too difficult. I typically only learn by example. So if at all possible, would you mind making a tutorial on how to create mobile game servers for dead/dying games using magiareco as an example case? Thank you!

error codes/download straight up impossible

really weird stuff when first opening and downloading assets
when using the special apk i get a few error codes(once every 5 minutes it seems to restart- shows error code 0, when i click retry it keeps going) and then the thing completely stops working at around 20k/37k (shows error code 110, at this point it wont download any more assets even if i try to retry it just give me the error again, these error codes say "Unable to access network. Do you want to retry?")
if i try to use an apk from apkpure it just goes to the gallery menu when 1000 assets are downloaded and that's it.
i'm pretty sure i got the certificate and everything done correctly, i tried using a dns server/non dns server, building the current repository and using that/using the latest release. i tried this both on my phone and on memu emulator

Bluestacks Android 7 rooted: Black Screen

Hey guys, i am trying to set the private server on my PC, and play via Bluestacks. I've rooted it, fired up the server, changed DNS to my IP, went to the site, installed the cert using a cert installer (cert installation seems to be broken in Bluestacks), rebooted the emulator just to be sure, and i am greeted with a black screen, both the official apk and the one that is created on the server's directory.

Can't access Android APK

The link on magica-us seems like it's broken? It gives me a 404 when I try to follow it. Is there some other place I can download it?

Should probably move `data` directory out of version control

Right now upgrading through git is a pain, because it always complains that files under data/ have been modified and prevents the git upgrade from continuing.

Here's a suggestion for how to deal with this:

• rename the current data/ to something else, say, data.template/
• when setting up the server or running it for the first time, if data/ does not exist, copy the contents of data.template/ to data/
• put data/ into .gitignore (since now the data/ directory will contain files that are being modified by the active server)

hope this makes sense, it is really late and I am tired lol

Can't get setupServer.bat to create ca.crt

I'm on windows and have been trying multiple times and, no matter how many times I run setupServer.bat, I cannot get ca.crt to be created. Without that, I cannot install it onto my phone.
Just to see what would happen, I tried launching startServer.bat. It simply gave an error message that seemed to imply that certain files should be in a certain location that weren't.

How can I get ca.crt to be created and for the startServer to work correctly?

Mirrors error: Missing file

Mirrors was working in the v 0.4 update, but now gives an error [500]. The command line outputs FileNotFoundError: [Errno 2] No such file or directory: 'data/user/userArenaBattleResult.json'

On a possibly unrelated note, I got this message on boot: "InsecureRequestWarning: Unverified HTTPS request is being made to host 'endless.snaa.services'. Adding certificate verification is strongly advised." Not sure what that means, or if it matters.

setupServer.sh misconfigured

Having set up a NUC with Linux for unrelated reasons, I thought I'd set this up for fun as well. Unfortunately, it didn't go very well at first, and while it's still not going very well, I can at least provide some information on what went wrong for later.

default: *** BEGINNING ZIPZAP VM SETUP ***
default:
default: # Installing packages...
default:
default: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
default: 5 packages can be upgraded. Run 'apt list --upgradable' to see them.
default:
default: # Disabling Ubuntu DNS manager...
default: Removed /etc/systemd/system/multi-user.target.wants/systemd-resolved.service.
default: Removed /etc/systemd/system/dbus-org.freedesktop.resolve1.service.
default:
default: # Setting up nginx web server in proxy mode...
default: # Generating SSL certificate...
default: python3: can't open file '../generate_cert.py': [Errno 2] No such file or directory
default: # Installing SSL certificate and starting up nginx...
default: cp: cannot stat '/ZipZap/ssl/ca.crt': No such file or directory
default: cp: cannot stat '/ZipZap/ssl/site.crt': No such file or directory
default: cp: cannot stat '/ZipZap/ssl/site.key': No such file or directory
default: Job for nginx.service failed because the control process exited with error code.
default: See "systemctl status nginx.service" and "journalctl -xe" for details.
default:
default: # Installing ZipZap python dependencies...
default: # Enabling ZipZap autostart at boot...
default: Created symlink /etc/systemd/system/multi-user.target.wants/rc-local.service → /etc/systemd/system/rc-local.service.
default:
default: # Starting ZipZap...
default:
default: *** ALL DONE! ***
default: This VM's public IP is: 192.168.1.244
default: Set your device/emulator's DNS to that address.
default: IP address is 192.168.1.244
default: Updating DNS 

This was after a git clone, but the same thing happened with the 0.5.3 source code release. (As an aside, the APK is only clearly placed in the Windows release zip...) I thought it might be an issue with the path of generate_cert.py being '../' instead of './', but that didn't fix it.

Further investigation revealed the problem was twofold. First, the ZipZap folder was mounted at /ZipZap, while the working directory remained at /home/vagrant (aka ~.). So I fixed that by changing the working directory. Second, the relative path to the python script to generate certs was still incorrect anyway, so I fixed that too.

I've already edited the file and will make a pull request to fix this, but I can't say for certain whether it actually totally works as intended now. For instance, I keep getting network errors of 1, 10, and 110 while trying to download the tutorial content (which is downloading extremely slowly either way). Switching to native Windows makes it work more smoothly, although not by much. There's still the occasional error when downloading, and the speed is still glacial on 5 GHz wifi that regularly handles much higher speeds.

Either there's some way of downloading the assets faster I'm not aware of, or I might recommend trying to find some way of creating a persistent cache or getting the files on the phone ahead of time. (Admittedly, I don't really know where the files are being pulled from in the first place...) The cache files also consistently only go up to less than 5 MB for the largest, and this is consistent on both the Vagrant VM and the Windows machine. (I've actually yet to get into the game at all, it's taken so long!)

Thus, I can't actually tell if the Vagrant version is working past simply letting me connect, get the cert, and download the files as expected.

If it matters - this is a fresh install! Anything I could do to make it easier for myself and a couple of friends to install this in a timely fashion would help a lot; either a persistent local cache, or a way of installing the files to your phone beforehand, would work wonders.

Need some tweaks in order to get the server running

I was trying the latest v0.3.2 release, but after following all steps in README, the app either got stuck in black screen or launched into gallery mode. After some research, I find that I have to do the following two tweaks in order to run the server successfully:

First, I have to comment out the following lines in nginx.conf:

    server {
        listen       443 ssl http2 default_server;
        return 503;
    }

Otherwise, I will always get 503 Service Temporarily Unavailable error whenever I visit https://android.magica-us.com in a web browser. This happens both on my mobile phone (after installing ca.crt) and the computer hosting nginx itself. After commenting out the above lines, I can access the "Welcome to nginx" page successfully.

Second, I have to move the installed user CA to system store, as I have described in #11. This means that I need to use a rooted device or emulator.

After making these two tweaks, I am able to connect to the private server successfully. I am unsure whether these tweaks are specific to my case, but I would like to open this issue so that developers might look into it.

setupServer.bat does not make a certificate

As the title says, running setupServer.bat like it says to in the instructions does not create a certificate file. If I append "pause" to the end of it to see the error, it says the following.

Unplayable battles

Hello!

Well, after playing the game for a while, I managed to play through almost* all the battles from Main Story and Another Story, as well as some Magical Girl Stories (but not all of them). Aside from some videos getting crashed (and thus making impossible to continue to read the whole story - specially the ending ones, that was sad :( ), ZipZap is overall a good experience for those who (like me) didn't have a chance to play the original NA server.

Aside from that, there are two battles which are unplayable, probably because they are meant to be played without a Support (that is, with a team containing Iroha alone).

Battles:

• Ch 3, Ep 7, Battle 4
• Ch 6, Ep 6, Battle 4

Since the server expects a support, it leads to a KeyError exception.

* The only battle I didn't manage to finish is the one which Walpurgisnacht has around 100.000.000 HP or something like that. Whenever I feel masochistic enough, I may try to beat it.

It crashes every time I try to do anything

Every time I try to access Stages, Mirrors, magical girl list, gacha, individual tabs within the archive, etc, I get the error "An unexpected error occurred. We are sorry for the inconvenience.[500]" The server command line reads, upon attempting to access the magical girls page:

KeyError: 400011
127.0.0.1 - - [2021-07-26 21:24:39] "GET /page/CharaListTop?value=userChapterList,userSectionList,userQuestBattleList&timeStamp=1627352678376 HTTP/1.1" 500 426 0.007001

Attempting to access Mirrors yields:

KeyError: 4000111
127.0.0.1 - - [2021-07-26 21:25:34] "GET /page/ArenaTop?value=userArenaBattle,userQuestBattleList,userSectionList&timeStamp=1627352732965 HTTP/1.1" 500 426 0.020001

and so on. This just started happening when I updated to the latest version. Everything on the main menu looks okay, though. I even got a daily login bonus. Any idea what's going on?

Uncertain Readme

Hello, this isn't an issue pre-sei but the tutorial in the readme stops after setting up the server but has no links for a client program to actually connect to the server. the https://android.magica-us.com link appears to have an upload option and some options starting with "fix scrollbug" but no downloadable APK or a download link labeled "ca.crt".

Is anyone able to explain how to proceed from here?

Thank you for your time.

Error Trying to level up Memorias

Hi. I don't know if you want things like issues on things as hard as this project, but I will leave it as just a way to make it better for community. Feel free to ignore it if you want.

So, I tried to level up a memoria (to make memoria space), and then an error arise on console. (On app, say something like "Connection Error, Reload").

Error point to api\userPiece.py. Line 204. Then go to Line 110 (Limit Break, idk why) and says userPiece['piece']['pieceKind']=='LIMIT_BREAK' 'pieceKind': Wrong Key

Is important to note I'm using memoria enhance, not memoria ascend, and using some 1* memoria (The familiar memos).

Thanks for your big efforts on this proyect.

Possible solution to Android stuck on black screen

It seems that a number of Android users are stuck on black screen and unable to reach the title screen, despite following all the required steps. ProxyDroid might be a solution, but in my particular case it didn't work either. After some research, I finally made it work, so I share my steps here. Note that the following steps require a rooted device or emulator (I use Android-x86 in VirtualBox).

0. Start server according to README.
1. On the client, install ProxyDroid, configure the proxy ip address and port, and start proxy.
2. Visit mitm.it, download and install the user CA according to README.
3. With root access, move the installed user CA to system CA by this guide.
4. Reboot your device or emulator.
5. After reboot, start ProxyDroid before starting server. Now launch the game and you should be able to connect to your private server.

In my case, both step 3 and 4 are necessary. Hope this will help other users with similar issues.

And last, a huge thanks to all developers involved!