Coder Social home page Coder Social logo

base58-universal's People

Contributors

davidlehn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

base58-universal's Issues

Explore possible optimizations.

The encode and decode time per element increases as the number of input elements grows. Looking at the output from a benchmark, it looks like it's mostly linear or super linear growth up to 16k elements.

Possible optimizations should be explored for use cases that use larger input sizes. This library uses a generic base-n decoder, so perhaps there are specific base58 optimizations. It might be good to also document this issue in the README so there are not performance surprises compared to simpler base16 and base64 codecs.

Some references (thanks to @msporny):
https://stackoverflow.com/questions/28418332/computational-complexity-of-base-conversion
https://cs.stackexchange.com/questions/21736/time-complexity-of-base-conversion

Benchmark and output:
#5
t1-16k.csv
t1-16k

ES6 import in Node 14 does not work

import {encode, decode} from 'base58-universal';
import {encode, decode} from 'base58-universal';
                ^^^^^^
SyntaxError: Named export 'decode' not found. The requested module 'base58-universal' is a CommonJS module, which may not support all module.exports as named exports.
CommonJS modules can always be imported via the default export, for example using:

import pkg from 'base58-universal';
const {encode, decode} = pkg;
import {encode} from 'base58-universal';
        ^^^^^^
SyntaxError: Named export 'encode' not found. The requested module 'base58-universal' is a CommonJS module, which may not support all module.exports as named exports.
CommonJS modules can always be imported via the default export, for example using:

import pkg from 'base58-universal';
const {encode} = pkg;

    at ModuleJob._instantiate (node:internal/modules/esm/module_job:104:21)
    at async ModuleJob.run (node:internal/modules/esm/module_job:149:5)
    at async Loader.import (node:internal/modules/esm/loader:166:24)
    at async Object.loadESM (node:internal/process/esm_loader:68:5)
import bs58 from 'base58-universal';
 console.log({bs58, encode: bs58.encode, decode: bs58.decode});
{ bs58: {}, encode: undefined, decode: undefined }
 import * as bs58 from 'base58-universal';
 console.log({bs58, encode: bs58.encode, decode: bs58.decode});
{
  bs58: [Module] { default: {} },
  encode: undefined,
  decode: undefined
}
node -v 
v14.15.3

Base58 encoding a ~366KB byte array takes relatively forever

I'm curious why encoding a byte array of only a few hundred kilobytes results in an operation that takes 10+ minutes? I actually can't find any materials that explains why base58 encoding would struggle with a moderate amount of data. Maybe someone here can explain?

Below is some example code

/**
 * nodejs script to test encrypting + encoding a base64 image data uri
 */

import fs from 'fs';
import * as url from 'url';
import path from 'path';
import { randomBytes, createCipheriv, createDecipheriv } from 'crypto';
import {encode, decode} from 'base58-universal';

import base58 from 'bs58';

const __dirname = url.fileURLToPath(new URL('.', import.meta.url));

const data = fs.readFileSync(path.join(__dirname, 'examples/docImageBase64String.txt'));

const key = randomBytes(32);
const iv = randomBytes(16);
const algorithm = 'aes-256-cbc';
const cipher = createCipheriv(algorithm, key, iv);

const startEncryption = new Date().getTime();

const encrypted1 = cipher.update(data);
const encrypted2 = cipher.final();
const encrypted = Buffer.concat([encrypted1, encrypted2]);

const endEncryption = new Date().getTime();

const timeToEncrypt = endEncryption - startEncryption;

console.log('time to encrypt', timeToEncrypt);

const beforeBase64Encoding = new Date().getTime();
const encodedBase64 = encrypted.toString('base64');
const afterBase64Encoding = new Date().getTime();

const timeToBase64Encode = afterBase64Encoding - beforeBase64Encoding;
console.log('time to encode base64 (using Buffer.toString())', timeToBase64Encode);

// write the encrypted data to a file so we can look at it without crashing the terminal lol
const outputDir = path.join(__dirname, 'output');

if (!fs.existsSync(outputDir)) {
  fs.mkdirSync(outputDir);
}
fs.writeFileSync(path.join(__dirname, 'output/encrypted.txt'), encodedBase64);

// base58 encoding
// this is V E R Y slow
const beforeEncoding = new Date().getTime();
const encoded = encode(encrypted);

const afterEncoding = new Date().getTime();
const timeToEncode = afterEncoding - beforeEncoding;
console.log('time to encode base58', timeToEncode);

// write base58 encoded data to a file
fs.writeFileSync('base58encoded.txt', encoded);

// let's decrypt it and write back to a file as a further sanity check
const decipher = createDecipheriv(algorithm, key, iv);

const decrypted1 = decipher.update(encrypted);
const decrypted2 = decipher.final();
const decrypted = Buffer.concat([decrypted1, decrypted2]);

fs.writeFileSync(path.join(__dirname, 'output/decrypted.txt'), decrypted.toString());

You'll just need to have a base64 encoded image in the /examples dir to run.

Running the above results in the output:

time to encrypt 2
time to encode base64 (using Buffer.toString()) 1
time to encode base58 1140551

However if one does run you'll see the base64 encoding taking a couple of ms while the base58 encoding takes tens of minutes. Why?

Thanks for your responses and for your efforts in open sourcing this project.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.