nucleuscloud / neosync

Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.

Open Source Data Anonymization and Synthetic Data Orchestration

| Website | Docs | Discord | Blog | Changelog | Roadmap

Introduction

Neosync is an open-source, developer-first way to anonymize PII, generate synthetic data and sync environments for better testing, debugging and developer experience.

Companies use Neosync to:

1. Safely test code against production data - Anonymize sensitive production data in order to safely use it locally for a better testing and developer experience
2. Easily reproduce production bugs locally - Anonymize and subset production data to get a safe, representative data set that you can use to locally reproduce production bugs quickly and efficiently
3. High quality data for lower-level environments - Catch bugs before they hit production when you hydrate your staging and QA environments with production-like data
4. Solve GDPR, DPDP, FERPA, HIPAA and more - Use anonymized and synthetic data to reduce your compliance scope and easily comply with laws like HIPAA, GDPR, and DPDP
5. Seed development databases - Easily seed development databases with synthetic data for unit testing, demos and more

Features

• Generate synthetic data based on your schema
• Anonymize existing production-data for a better developer experience
• Subset your production database for local and CI testing using any SQL query
• Complete async pipeline that automatically handles job retries, failures and playback using an event-sourcing model
• Referential integrity for your data automatically
• Declarative, GitOps based configs as a step in your CI pipeline to hydrate your CI DB
• Pre-built data transformers for all major data types
• Custom data transformers using javascript or LLMs
• Pre-built integrations with Postgres, Mysql, S3

Getting started

Neosync is a fully dockerized setup which makes it easy to get up and running.

A compose.yml file at the root contains production image refs that allow you to get up and running with just a few commands without having to build anything on your system.

Neosync uses the newer docker compose command, so be sure to have that installed on your machine.

To start Neosync, clone the repo into a local directory, be sure to have docker installed and running, and then run:

make compose/up

To stop, run:

make compose/down

Neosync will now be available on http://localhost:3000.

The production compose pre-seeds with connections and jobs to get you started! Simply run the generate and sync job to watch Neosync in action!

Kubernetes, Auth Mode and more

For more in-depth details on environment variables, Kubernetes deployments, and a production-ready guide, check out the Deploy Neosync section of our Docs.

Resources

Some resources to help you along the way:

• Docs for comprehensive documentation and guides
• Discord for discussion with the community and Neosync team
• X for the latest updates

Contributing

We love contributions big and small. Here are just a few ways that you can contribute to Neosync.

• Join our Discord channel and ask us any questions there
• Open a PR (see our instructions on developing with Neosync locally)
• Submit a feature request or bug report

Licensing

We strongly believe in free and open source software and make this repo is available under the MIT expat license.

neosync's Issues

[NEOS-477] Publish TypeScript SDK for Neosync

_{From SyncLinear.com | NEOS-477}

[NEOS-472] support passing in seed values for transformers to get consistent results

_{From SyncLinear.com | NEOS-472}

[NEOS-509] Neosync logo gets smaller when page is half size

Logo seems to scale to the size of the window.

It shouldn't do that 😃

_{From SyncLinear.com | NEOS-509}

[NEOS-430] launch overview page

_{From SyncLinear.com | NEOS-430}

[NEOS-438] publish public facing API and SDKs

_{From SyncLinear.com | NEOS-438}

[NEOS-442] Update job status

job status currently doesn't support enough states and map correctly to temporal.

1. Remove status field from Jobs db table.
2. Map Neosync status to temporal status (pauseStatus, scheduleStatus)
3. Update FE to support pause status and schedule status

_{From SyncLinear.com | NEOS-442}

[NEOS-457] Bug with user not in account

This happened when alisha switched the database out.

the frontend was trying to update the session storage to include the latest account but it kept being overwritten to the old one.

This is doubled by the fact that SWR keps trying to reach out to the jobs api and was met with a 403. It was doing it endlessly though. Why was it not backing off? Need to look into this.

_{From SyncLinear.com | NEOS-457}

[NEOS-470] Enable auth token refreshing in CLI

_{From SyncLinear.com | NEOS-470}

[NEOS-491] add length param to random float transformer

right now the random float transformer returns 17-18 digits for every number which might be too much precision for some use-cases.

someone should be able to specify a length param that the returned value should abide by

_{From SyncLinear.com | NEOS-491}

[NEOS-436] re-work auth page ux

_{From SyncLinear.com | NEOS-436}

[NEOS-461] Callout npm install prior to docker compose up

We do this now for go but need to do the same thing for npm.

we could also add a top level makefile to make this easier

we could also add all of this stuff to the dev dockerifles to make it more seamless.

_{From SyncLinear.com | NEOS-461}

[NEOS-500] Clicking home should route to home page for currently selected account

we could either render the home button as the absolute link, or persist the last known selected account. I kinda like the home button always being the base url, then letting the home nav take you to the jobs page of the last known selected account. that is more complex however.

_{From SyncLinear.com | NEOS-500}

[NEOS-465] Job Run should only allow Cancellation if the run is active

The action Cancel on the list page as well as "Cancel Run' button on the run details page is always there regardless of the run status.

We should be smarter here and only show this if the run is active.

_{From SyncLinear.com | NEOS-465}

[NEOS-485] update transformers to take out defaults for length-based transformers

• strings, ints, floats, etc.
• handle nils for these values in the transformers, if the user doesn't pass anything in, we should just randomize between the min and max

_{From SyncLinear.com | NEOS-485}

[NEOS-462] Create API Interceptor that logs start/end request

We need an interceptor that logs start and requests but that is built for connect.

Should be able to adapt it from this: https://github.com/grpc-ecosystem/go-grpc-middleware/tree/main/interceptors/logging

_{From SyncLinear.com | NEOS-462}

[NEOS-449] Add Docs for how to install Neosync CLI

We could use some docs for installing neosync cli.

we have some pretty good ones over at the old nucleuscloud cli docs, we could use that as a starting template in terms of layout and what content we should expect.

https://docs.nucleuscloud.com/home/getting-started/installing-the-nucleus-cli

_{From SyncLinear.com | NEOS-449}

[NEOS-499] For consistency, always route /account/settings to temporal page

right now if it's a team account we route to members. this is kinda weird and we should just always route to the temporal page for consistency.

_{From SyncLinear.com | NEOS-499}

[NEOS-441] Reduce Auth0 Dependency on Frontend and Backend

Both the app and api are pretty dependent on auth0. We should make this instead generic for oidc providers.

_{From SyncLinear.com | NEOS-441}

[NEOS-424] new public issue - test evis

_{From SyncLinear.com | NEOS-424}

[NEOS-429] test-evis-issue

_{From SyncLinear.com | NEOS-429}

[NEOS-445] Write Docs for setting up Authentication

_{From SyncLinear.com | NEOS-445}

[NEOS-434] Add Endpoint to show what release the API is currently at

Similar to the metadata we build into the CLI, we should do the same for the backend.

It would be great to then expose this in the App or API somehow.

Should this be authenticated if auth=true?

_{From SyncLinear.com | NEOS-434}

[NEOS-496] Add support for Google Auth

It's probably good we support more than one auth provider to keep ourselves honest.

Today we only support Auth0 in neosync. We should also support Google

_{From SyncLinear.com | NEOS-496}

[NEOS-458] Investigate Database Docker Volume missing file and permission issues

We saw that when we mounted the postgres db volume for neosync-db there were missing files and missing permissions.

The problems went away when removing the passthrough volume mount.

Why is temporal not suffering from this?

_{From SyncLinear.com | NEOS-458}

[NEOS-451] Lists are not properly rendered in docs

Both ordered and unordered lists do not have a bullet or numbers rendered in them.

We should fix this.

_{From SyncLinear.com | NEOS-451}

[NEOS-453] docs on mobile aren't rendering correctly

need to update margins and check styling

_{From SyncLinear.com | NEOS-453}

[NEOS-497] Auth isn't working quite right on Staging

I havent been able to reproduce this on localhost, just staging for some reason.

account switcher goes away when you click on the home button (but none of the other pages?)

auth redirect doesnt always happen

Clicking around enough eventually logs you in.

need to figure out what the deal is here

Home page seems to be totally broken

_{From SyncLinear.com | NEOS-497}

[NEOS-466] Remove Job.ConnectionSourceId from Database Schema

We no longer use this and is a source of bugs as it is still present on the database model

_{From SyncLinear.com | NEOS-466}

bare metal and fly.io

Is your feature request related to a problem?

docs refer to baremetal and would like this.

Describe the solution you'd like

I like to use Fly.io, which uses a single dockerFile.

https://fly.io/docs/app-guides/multiple-processes/

So what we do is as part of the Docker boiler, you stuff in your binaries, which in neosync 2 or 3 I think.
Fly has the concept of Apps, and it will boost each App ( which is a binary ).

The other way that we can use Overwind, to run the binaries outside docker or inside docker.

https://github.com/DarthSim/overmind

Describe alternatives you've considered

Additional context

https://fly.io/docs/hands-on/install-flyctl/ is golang.

Fly can scale to zero and also scale per region
https://fly.io/docs/apps/scale-count/#:~:text=Scale%20an%20app%27s%20regions,its%20changes%20across%20these%20regions.

Thank you for your feature request – we love each and every one!

[NEOS-440] data gen recent job runs is empty

weird that it has a started at value but nothing for the status?

_{From SyncLinear.com | NEOS-440}

[NEOS-381] make system transformers an enum in proto so they're statically typed

_{From SyncLinear.com | NEOS-381}

[NEOS-450] Write docs for basic SDK usage

_{From SyncLinear.com | NEOS-450}

[NEOS-435] implement posthog for analytics

_{From SyncLinear.com | NEOS-435}

[NEOS-460] [NEOS-459] Update readme to include better callout for mac docker desktop file mounts

we mount volumes to the filesystem so it's imperative that this gets safelisted in mac docker desktop otherwise it doesnt work.

_{From SyncLinear.com | NEOS-459}

_NEOS-460

[NEOS-425] new test issue board

_{From SyncLinear.com | NEOS-425}

[NEOS-508] Reduce Auth0 Requirement on Frontend

This is already pretty minimal thanks to next-auth.

However, we load in specific Auth0 named env vars today.

We should make this more generic and just ask for one set.

Next-auth technically supports multiple providers, but usually an OIDC provider is used as the front for those.

_{From SyncLinear.com | NEOS-508}

[NEOS-507] Refactor user_identity_provider_associations to be more generic

We want to reduce our dependency on auth0 in favor or any OIDC compliant provider.

To do so, we need to refactor this table to be more generic.

Perhaps the new table should follow this schema: provider_name, provider_identifier

The auth0_provider_id column would be replaced by the following:

provider_name | provider_identifier
----------------------------------
        auth0 | google|[email protected]
        github| nickzelei
        google| [email protected]

_{From SyncLinear.com | NEOS-507}

[NEOS-469] Make transformer name column clickable in /transformers table

Might as well make all of the resources name clickable. most other tables have a uuid col and that is clickable.

_{From SyncLinear.com | NEOS-469}

[NEOS-501] Add TS SDK Docs

_{From SyncLinear.com | NEOS-501}

[NEOS-454] Add docs to all proto files

_{From SyncLinear.com | NEOS-454}

[NEOS-483] API Key component showing all the time

The input box and copy button should only show when we have the api key contents

This is the /<team>/settings/api-keys/:id page

_{From SyncLinear.com | NEOS-483}

[NEOS-479] Add CI for TS SDK

Need to publish this package upon release

_{From SyncLinear.com | NEOS-479}

[NEOS-433] add guides section to docs

write guides for:

• using the neosync github action and cli to sync CI databases
• creating a job
• configuring authentication

_{From SyncLinear.com | NEOS-433}

[NEOS-503] Add neosync logout CLI command

Gives the ability to logout. Should clean up access/refresh tokens on disk if they are present.

This should also request a logout url from the server and open a web browser to log the user out of their session.

Check out similar logic in the nucleuscloud cli: https://github.com/nucleuscloud/cli/blob/main/cmd/logout.go

_{From SyncLinear.com | NEOS-503}

[NEOS-467] Add Account Context to CLI

Need a way to set account id in CLI.

Either as context or as flag.

trigger, list, sync

_{From SyncLinear.com | NEOS-467}

[NEOS-498] When on a team account, clicking on Settings routes you back to your personal account

The link is correct, but somehow you are re-routed back to the default account.

Must have something to do with this useeffect:

  useEffect(() => {
    if (isSystemAppConfigDataLoading || isAccountLoading) {
      return;
    }
    if (systemAppConfigData?.isAuthEnabled && account?.name) {
      return router.push(`/${account?.name}/settings/members`);
    } else {
      return router.push('/personal/settings/temporal');
    }
  }, [
    account?.id,
    isAccountLoading,
    systemAppConfigData?.isAuthEnabled,
    isSystemAppConfigDataLoading,
  ]);

_{From SyncLinear.com | NEOS-498}

[NEOS-447] Docs should have a dark mode

Reading neosync docs at night is blinding and we should support dark mode for them.

_{From SyncLinear.com | NEOS-447}

[NEOS-416] Test2asdfdsf

this is to test the github<>linear integration

_{From SyncLinear.com | NEOS-416}

[NEOS-423] Add buf fmt action to fail if diff

we shouldnt allow malformatted protos.

I thought buf lint did this but it doesn't seem to care if buf fmt returns changes.

_{From SyncLinear.com | NEOS-423}

[NEOS-431] review dark mode stylings

_{From SyncLinear.com | NEOS-431}