papyrus-mc / papyrusjs Goto Github PK

[mcbserver@c ~]$ ./papyrus --world="./serverfiles/worlds/Bedrock level" --textures="./Vanilla_Resource_Pack" --output="./worldmap" --mode="topdown_shaded" --force-download papyrus.js v1.0.5a by clarkx86 Threads: 4 Texture directory is missing or --force-download has been specified. Downloading... /home/mcbserver/Vanilla_Resource_Pack/textures.tmp.zip [=================================] 100%DownloadDownloading...2846205813/ 46205813 Extracting... (node:29866) UnhandledPromiseRejectionWarning: Error: Failed to open database at MinecraftLevelDOWN._open (/snapshot/papyrusjs/node_modules/leveldb-mcpe/index.js:39:19) at MinecraftLevelDOWN.AbstractLevelDOWN.open (/snapshot/papyrusjs/node_modules/abstract-leveldown/abstract-leveldown.js:36:10) at DeferredLevelDOWN._open (/snapshot/papyrusjs/node_modules/deferred-leveldown/deferred-leveldown.js:19:12) at DeferredLevelDOWN.AbstractLevelDOWN.open (/snapshot/papyrusjs/node_modules/deferred-leveldown/node_modules/abstract-leveldown/abstract-leveldown.js:38:8) at LevelUP.open (/snapshot/papyrusjs/node_modules/levelup/lib/levelup.js:89:11) at new LevelUP (/snapshot/papyrusjs/node_modules/levelup/lib/levelup.js:61:8) at LevelUP (/snapshot/papyrusjs/node_modules/levelup/lib/levelup.js:33:12) at init (/snapshot/papyrusjs/app.js:0:0) at Promise.then (/snapshot/papyrusjs/app.js:0:0) (node:29866) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1) (node:29866) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Hi,

I run a server with 2GB ram, when papyrusjs generate zoom level for my map, the process is killed.
The world is 22MB.
In syslog file, it shows:

Out of memory: Kill process 27511 (node) score 861 or sacrifice child
May 27 10:22:09 vps-01745824 kernel: [246841.076866] Killed process 27511 (node) total-vm:2494560kB, anon-rss:1710500kB, file-rss:0kB, shmem-rss:0kB

I found the solution to this problem:
1/ Add these lines in renderZoomLevel.js

diff --git a/render/renderZoomLevel.js b/render/renderZoomLevel.js
index 6e958f6..b3998d9 100644
--- a/render/renderZoomLevel.js
+++ b/render/renderZoomLevel.js
@@ -76,6 +76,8 @@ module.exports = function (chunkSize, zoomLevelMax, chunkX, chunkZ) {
                                     fs.mkdirSync(path.normalize(path_output + '/map/' + (zoomLevelCurrent) + '/' + (ix) + '/'));
                                 }
                                 tileImg.saveSync(path.normalize(path_output + '/map/' + (zoomLevelCurrent) + '/' + (ix) + '/' + (iz) + '.png'));
+                               tileImg = null;
+                               global.gc();
                                 resolve();
                             }
                             )

2/ Call node with arg: --expose-gc

With this setting, I ran map generation with no problem, using max 500MB memory
Warning! If node is not launched with --expose-gc, it will crash on this step.

With PR #15 Windows support is now possible in theory. Quoting the PR summary:

The problem is that Mapnik dropped Windows support a while ago and I've to find a library that is nearly as fast as Mapnik for stitching images.

If anyone has a good suggestion for a comparably fast module such as Mapnik that compiles on Linux, Windows, macOS x86 & x64 (and optimally ARM), I would be more than happy to investigate and implement it.

As a kick start, my benchmarks showed that jimp is way to slow for this use case and sharp is currently not nearly as fast as Mapnik for stitching images.

Thanks,
clarkx86 :-)

Extracting...
Attempting to open database...
Success!
Reading database. This can take a couple of seconds up to a couple of minutes.
World reaches from chunks X: -43, Z: -48 to X: +96, Z: +82
Processing and rendering 4213 Chunks, which 21959 of them are valid SubChunks...
[--------------------------------] 0% Processing chunk 3/ 4213 31 chunks/Second(node:31220) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'sand' of null
at loadTexture (/snapshot/papyrusjs/render/loadTexture.js:25:23)
at render (/snapshot/papyrusjs/render/renderChunk.js:44:47)
at /snapshot/papyrusjs/render/renderChunk.js:22:9
at new Promise ()
at /snapshot/papyrusjs/render/renderChunk.js:12:12
at process. (/snapshot/papyrusjs/app.js:360:17)
at process.emit (events.js:215:7)
at emit (internal/child_process.js:876:12)
at processTicksAndRejections (internal/process/task_queues.js:81:21)
(node:31220) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:31220) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
[--------------------------------] 0% Processing chunk 4/ 4213 34 chunks/Second(node:31214) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'sand' of null
at loadTexture (/snapshot/papyrusjs/render/loadTexture.js:25:23)
at render (/snapshot/papyrusjs/render/renderChunk.js:44:47)
at /snapshot/papyrusjs/render/renderChunk.js:22:9
at new Promise ()
at /snapshot/papyrusjs/render/renderChunk.js:12:12
at process. (/snapshot/papyrusjs/app.js:360:17)
at process.emit (events.js:215:7)
at emit (internal/child_process.js:876:12)
at processTicksAndRejections (internal/process/task_queues.js:81:21)
(node:31214) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:31214) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:31208) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'sand' of null
at loadTexture (/snapshot/papyrusjs/render/loadTexture.js:25:23)
at render (/snapshot/papyrusjs/render/renderChunk.js:44:47)
at /snapshot/papyrusjs/render/renderChunk.js:22:9
at new Promise ()
at /snapshot/papyrusjs/render/renderChunk.js:12:12
at process. (/snapshot/papyrusjs/app.js:360:17)
at process.emit (events.js:215:7)
at emit (internal/child_process.js:876:12)
at processTicksAndRejections (internal/process/task_queues.js:81:21)
(node:31208) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:31208) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:31226) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'stone' of null
at loadTexture (/snapshot/papyrusjs/render/loadTexture.js:25:23)
at render (/snapshot/papyrusjs/render/renderChunk.js:44:47)
at /snapshot/papyrusjs/render/renderChunk.js:22:9
at new Promise ()
at /snapshot/papyrusjs/render/renderChunk.js:12:12
at process. (/snapshot/papyrusjs/app.js:360:17)
at process.emit (events.js:215:7)
at emit (internal/child_process.js:876:12)
at processTicksAndRejections (internal/process/task_queues.js:81:21)
(node:31226) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:31226) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Missing texture mappings for data values for still_lava, log, log2, composter_top, farmland, beetroot, carrots, potatoes, sapling, pumpkin_top, pumpkin_stem, chest_inventory_top, leaves, and planks.

Minecraft Windows 10 1.11.4
Map file copied from the Worlds folder of Minecraft Alpha (Bedrock)
Ubuntu 16.04
Apache/2.4.18 (Ubuntu)
NodeJS v8.16.0

ubuntu@ubuntu:/var/www/html$ sudo ./papyrus --world="./Bedrock/" --textures="./Vanilla_Resource_Pack" --output="./map" --mode="topdown_shaded"
papyrus.js v1.0.5a by clarkx86
Threads: 2
Texture directory is missing or --force-download has been specified. Downloading...
/var/www/html/Vanilla_Resource_Pack/textures.tmp.zip
[=================================] 100%DownloadDownloading...7245632250/ 45632250
Extracting...
Reading database. This can take a couple of seconds up to a couple of minutes.
Creating Leaflet map...
Furthest X (negative):  0       Furthest X (positive):  341
Furthest Z (negative):  -83     Furthest Z (positive):  84
Processing and rendering 14412 Chunks, which 65993 of them are valid SubChunks...
[--------------------------------] 0%   Processing chunk 2/ 14412       4 chunks/Second(node:8996) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'water' of null
    at loadTexture (/snapshot/papyrusjs/src/render/loadTexture.js:0:0)
    at render (/snapshot/papyrusjs/src/render/renderChunk.js:0:0)
    at module.exports.Promise (/snapshot/papyrusjs/src/render/renderChunk.js:0:0)
    at new Promise (<anonymous>)
    at module.exports (/snapshot/papyrusjs/src/render/renderChunk.js:0:0)
    at process.on (/snapshot/papyrusjs/app.js:0:0)
    at process.emit (events.js:194:15)
    at emit (internal/child_process.js:820:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
(node:8996) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:8996) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
[--------------------------------] 0%   Processing chunk 3/ 14412       4 chunks/Second(node:9002) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'grass' of null
    at loadTexture (/snapshot/papyrusjs/src/render/loadTexture.js:0:0)
    at render (/snapshot/papyrusjs/src/render/renderChunk.js:0:0)
    at module.exports.Promise (/snapshot/papyrusjs/src/render/renderChunk.js:0:0)
    at new Promise (<anonymous>)
    at module.exports (/snapshot/papyrusjs/src/render/renderChunk.js:0:0)
    at process.on (/snapshot/papyrusjs/app.js:0:0)
    at process.emit (events.js:194:15)
    at emit (internal/child_process.js:820:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
(node:9002) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:9002) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

If the number of chunks is not a multiple of the number of render threads, then chunks will be skipped and strip like blanks will appear in the generated map.

papyrus.js currently has a bug while decoding SubChunks, where block states beyond the SubChunk's palette sizes get requested. This is simply ignored by the application, but throws an error for the user:

[WARNING] Palette ID out of bounds! 7 : 5

It is currently unclear if this is a problem of actually reading the database or a corrupt SubChunk.
Even after re-generating the world from the seed, the bug still occurs, but it kind of (?) depends if the world is a creative or surival world (not sure however).

I attached a .zip for a test world to reproduce this behaviour: TestWorld.zip

When some textures are selected, such as waterlilies, the texture file resolves to texture[object Object].tga. This prevents waterlilies from being added to the maps.

Hi, as the feature to incrementally render maps is still on the way, I would need to manually run this on my Mac. Unfortunately, I dont think that the binary would work (ye, I tried it) so im left with compiling from scratch.

I've tried installing the dependencies via:

brew install node cmake gcc 
sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

The last line being a way to install zlib (https://stackoverflow.com/questions/52741673/how-can-i-install-zlib-on-mac-os-x-mojave-10-14). Which then gets me:

$cmake --version
cmake version 3.15.0
CMake suite maintained and supported by Kitware (kitware.com/cmake).

g++ --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

But it seems that first im getting a lot of warnings about deprecations in MacOS 10.12

In file included from ./port/port_posix.h:47:
./port/atomic_pointer.h:64:3: warning: 'OSMemoryBarrier' is deprecated: first deprecated in macOS 10.12 - Use std::atomic_thread_fence() from <atomic> instead
      [-Wdeprecated-declarations]
  OSMemoryBarrier();
  ^

Then started getting C++11 related issues

In file included from /Users/pogz/Downloads/papyrusjs/node_modules/leveldb-mcpe/src/entrypoint.cpp:2:
/Users/pogz/Downloads/papyrusjs/node_modules/nan/nan.h:50:3: error: This version of node/NAN/v8 requires a C++11 compiler
# error This version of node/NAN/v8 requires a C++11 compiler
  ^

Anyone tried this successfully to compile on a Mac?

If you set --threads to something other than the number of reported CPUs the program does not reliably exit.

Getting this running latest alpha. With BDS version 1.14.32.2 on CentOS 8

./papyrusjs --world="./serverfiles/worlds/Bedrock level" --textures="./Vanilla_Resource_Pack" --output="./papyrus" --mode="topdown_shaded" papyrus.js v1.1.0a by clarkx86 and contributors Threads: 4 Texture directory is missing or --force-download has been specified. Downloading... /home/mcbserver/Vanilla_Resource_Pack/textures.tmp.zip [=================================] 100%DownloadDownloading...2846205813/ 46205813 Extracting... Attempting to open database... Success! Reading database. This can take a couple of seconds up to a couple of minutes. World reaches from chunks X: -100, Z: -164 to X: +157, Z: +196 Processing and rendering 21552 Chunks, which 106427 of them are valid SubChunks... [--------------------------------] 0% Processing chunk 4/ 21552 85 chunks/Second(node:5056) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'stone' of null at loadTexture (/snapshot/papyrusjs/render/loadTexture.js:25:23) at render (/snapshot/papyrusjs/render/renderChunk.js:44:47) at /snapshot/papyrusjs/render/renderChunk.js:22:9 at new Promise (<anonymous>) at /snapshot/papyrusjs/render/renderChunk.js:12:12 at process.<anonymous> (/snapshot/papyrusjs/app.js:360:17) at process.emit (events.js:215:7) at emit (internal/child_process.js:876:12) at processTicksAndRejections (internal/process/task_queues.js:81:21) (node:5056) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2) (node:5056) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. (node:5074) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'grass' of null at loadTexture (/snapshot/papyrusjs/render/loadTexture.js:25:23) at render (/snapshot/papyrusjs/render/renderChunk.js:44:47) at /snapshot/papyrusjs/render/renderChunk.js:22:9 at new Promise (<anonymous>) at /snapshot/papyrusjs/render/renderChunk.js:12:12 at process.<anonymous> (/snapshot/papyrusjs/app.js:360:17) at process.emit (events.js:215:7) at emit (internal/child_process.js:876:12) at processTicksAndRejections (internal/process/task_queues.js:81:21) (node:5074) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2) (node:5074) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. (node:5068) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'stone' of null at loadTexture (/snapshot/papyrusjs/render/loadTexture.js:25:23) at render (/snapshot/papyrusjs/render/renderChunk.js:44:47) at /snapshot/papyrusjs/render/renderChunk.js:22:9 at new Promise (<anonymous>) at /snapshot/papyrusjs/render/renderChunk.js:12:12 at process.<anonymous> (/snapshot/papyrusjs/app.js:360:17) at process.emit (events.js:215:7) at emit (internal/child_process.js:876:12) at processTicksAndRejections (internal/process/task_queues.js:81:21) (node:5068) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2) (node:5068) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. (node:5062) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'water' of null at loadTexture (/snapshot/papyrusjs/render/loadTexture.js:25:23) at render (/snapshot/papyrusjs/render/renderChunk.js:44:47) at /snapshot/papyrusjs/render/renderChunk.js:22:9 at new Promise (<anonymous>) at /snapshot/papyrusjs/render/renderChunk.js:12:12 at process.<anonymous> (/snapshot/papyrusjs/app.js:360:17) at process.emit (events.js:215:7) at emit (internal/child_process.js:876:12) at processTicksAndRejections (internal/process/task_queues.js:81:21) (node:5062) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2) (node:5062) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.