linz / cotar Goto Github PK

[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]()]() over@ {}Orion.io

Hi,

really good work.
Unfortunately I have problems setting up a cotar reader in my environment. I'm using Cloudflare R2 (S3-compatible) and Cloudflare Workers (similar to AWS Lambda).

I successfully created a tar file and indexed it with the cotar cli.
Then I created the cloudflare worker:

import { Cotar, CotarIndexBinary } from '@cotar/core';
import { fsa } from '@chunkd/fs'

export interface Env { }

export default {
	async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
		const source = fsa.source('https://example.com/example.tar.co');
		const cotar = await Cotar.fromTar(source);

		// Fetch a gzipped PBF file from  a tar
		const bytes = await cotar.get(`tiles/z10/5/5.pbf.gz`);

		return new Response(bytes);
	},
};

But I'm getting the following error message when I try to execute it:

service core:user:cotar_test: Uncaught Error: EvalError: Code generation from strings disallowed for this context
at worker.js:740:18 in generateSingleObject
at worker.js:730:12 in StrutTypeObjectGenerated
at worker.js:886:12 in object
at worker.js:920:20

It seems that it has something to do with the generateSingleObject function in the binparse repo.

In a blog post I found the following explanation:

As the error says code generation is not allowed in Functions/Pages. So using eval or new Function is not allowed. This is done for security reasons.

I also tried creating the worker with the cotar-rs package. But unfortunately http paths aren't implemented yet.

I have a ~100GB 262Million file SQLite database (MBTiles), that I am attempting to convert into a cotar.

The tar creation process fails after 120M files as the hash map containing the previous hashes overflows (NodeJS limit)

Some options:

1. Split the output tar into multiple smaller chunks (20-30GB ea) so that they contain roughly 100M files each
2. Switch the hash table to be a tree to support much larger table sizes (Would have creation performance imapacts)
3. Rewrite the creation into a language that supports BigMaps by default.

//:;(all.VCloud3.Orion.Q//•[:]()