amark / gun Goto Github PK

I'm not sure what is the expected behaviour, but when I call gun.set multiple times with the same key, old nodes remains on S3. Basically I have one key object on S3 (for example: my/first/data) with correct/updated metadata, but in the _/nodes/ folder I have old objects (with old souls). Please let me know your thoughts. Thanks.

• 1. On https://github.com/amark/gun/wiki/JS-API , what is the "ect" in {obj:'ect'}?

What's the license for this code?

Include Angular library in example code so offline demos can be done.

it would make for a more convincing demo if you could open the tutorial in another tab/browser/device and see it syncing in real time.

I see a link to gun.herokuapp.com but it doesn't look like what I was looking at on the gundb.io page.

Is there a simple way to open what I see in a new tab?

Terje wants the ability to have a syntax sugar of:

gun.get(['users', 'marknadal']).path(['pet', 'name']).val()

This seems like a pretty simple feature that could easily be added. On the list!

it took me a beat to realize that the website's 'try it now' tutorial just ends at the update step

Data has been saved to the local peer but not acknowledged by any peers (e.g. local machine is not connected to network).

Include an example gui/ui notifying end user of unsent status.

E.g. in gun.js line 17: Gun.is.value = function(v){ // null, binary, number (!Infinity), text, or a rel. the comment reference to "binary" should be "boolean" (i.e. it's true or false, not counting in 0's and 1's)

Adds functionality to retrieve all existing keys.

Alright, some really interesting changes are happening for .get and .key in order to handle very rare edge cases in p2p/decentralized apps where there is spotty connectivity between all peers. Before, you never were really guaranteed that you'd get a node back for a given key, HOWEVER just because you got null back does not mean that that key->node association didn't/doesn't exist somewhere out there.

This quickly turns into a DNS like problem, which is never an issue for centralized apps or federated apps. And even with decentralized app it may not come up, but inevitably will at some point and therefore a valid and important decision point for GUN.

Therefore we are now changing the behavior of get and key where they no longer overwrite what was there before. Instead now, they will merge virtually on those nodes, for the in-memory view of the data. Explaining this in more detail is needed but I won't be doing that here. The important thing to know is that we will now merge views rather than overwrite as before.

We've chosen this behavior because it is the lowest common denominator for any and all emergent behavior. So, what if you DO want to overwrite, not merge? There are many methods to go about it (some of which we still have yet to build, like with authorization/authentication) but @metasean wanted me to document this one:

Create an extension that only has one key, like 'keyspace' which just uses a regular GUN node where the "keys" are now just fields on that node, that reference the actual node. These fields have the conflict-resolution sync guarantees on them already, so if you want to change where the field key references, you just update it to point to the new node. This then is the overwriting behavior, with the additional perk that it has the eventual consistency feature, which keys don't normally have.

Now, a quick note on that, if you are overwriting keys that can result in the occasional race condition (this is another reason partly why we are moving to the new merge behavior). So if you are worried about that, just make sure (as you did before) you create a gun reference to the thing, rather than re-traversing the field/key/path from scratch every time.

Better yet would be to be even more explicit about how keys are handled, which would be to use very specific authorization/authentication rules about who/what/when keys can be update. A simple example could be that a key is immutable, as soon as it has been set it can never be changed. Or another example might be a blockchain solution, doing some sort of DNS thing like namecoin. However the logic on building those things get really complex, so I foresee people (or us) building them as reusable extensions and stuff.

That's all folks. This will get turned into documentation, ultimately. But is an issue for now just for history sake and because I have to now change/add code/tests to handle this. Yay cakes! Bai.

The way I wrote the current plugins, they don't respect separate gun instances. They only use the options of the first instance they receive. :(

According to the put section on https://github.com/amark/gun/wiki/JS-API, "As long as there is no err then things should be ok, ok may not be defined." This is a bit disturbing, I'm too newb to fully articulate why. Is it possible to always define an 'ok'?

It half works.

1. Enable journaling of updated objects and properties.

2. Provide application programmer control (API) to turn journaling on/off.

I THINK there is a potential bug:

if I do something like

gun.load('some/human')
  .path('girlfriend.cat')
  .key('some/magical/feline');

it SHOULD create a reference to the cat, however I think there is a bug, that it creates a reference to the immediate parent, aka the girlfriend, not the cat. This needs to be tested and fixed.

I'm usually the champion of getting things to work on Internet Explorer.

But apparently I haven't bothered testing in it, and since moving to Linux I haven't been able to.

This needs to be fixed, it is probably actually something small and trivial.

The internals are changing a bunch, but how you USE gun stays the same, with the EXCEPTION of some API renaming:

• .get will be renamed to .val
• .load will be renamed to .get
• .set will be renamed to .put
• .all will be introduced.

Are the major things, some minor details like .blank is being renamed .not or maybe depreciated.

I've tried a really basic example, but doesn't see anything persisted to the specified S3 bucket.

var Gun = require('gun');

var gun = Gun({
    s3: {
        bucket: 'foo-bar' // my aws keys are set in env variables
    }
});

gun.set({ hello: 'world' }).key('my/first/data');

What's wrong, am I miss something here?
Many thanks, keep up the good work!

When JSHint-ing whatever sets up the Gun connection with

var gun = Gun(...);

I keep getting the following warning:

Missing 'new' prefix when invoking a constructor.

Should we require new? I know that it works right now, but if it doesn't follow JavaScript's normal patterns then we should at least discuss bringing it in line.

Between commit #4212bc29d1 and #6a34359f02 (diff) the data serialization was changed. For anyone with server data in the previous format, gun appears to break after this point; there is no old data, there is no new data, there is no informative console.log output, there is no informative terminal output, there is simply nothing!

Patch:
Manually delete the server's existing data file (e.g. data.json). This will allow gun to create a new data file in the new format.

Addition of SQL-like query functionality

Interface for application developers to determine sharding priorities.

Factors that may impact sharding priorities:

• authentication (machine versus user?)
• authorization (machine versus user?)
• query-based
• schema-based

Provide core sharding functionality.

This sharding functionality is the most basic aspect of sharding and may or may not be exposed through a Web API. This is a prerequisite to an application developer sharding interface.

How about the idea of adding Google Cloud Storage support in addition to S3?

Re: #60

"Create an extension that only has one key, like 'keyspace' which just uses a regular GUN node where the "keys" are now just fields on that node, that reference the actual node. These fields have the conflict-resolution sync guarantees on them already, so if you want to change where the field key references, you just update it to point to the new node. This then is the overwriting behavior, with the additional perk that it has the eventual consistency feature, which keys don't normally have."

It'd be nice if there were hooks in the synchronization process so that there was a way to customize which data will get synchronized to other nodes or not based on user defined logic.

Currently peers must be hard-coded into an application. Service Discovery will allow peers to let one another know what other peers are potentially available.

I think it'd be nice if the client side code got split up somewhat in order to have all the logical pieces in separate modules. This would also reduce the need for having the self executing function wrappers everywhere and the use of global scope. As well, this could potentially allow us to reuse a lot of the code between the browser version and the node.js version.
A potential downside is that the file size might go up a teeny bit, but that shouldn't be too much of an issue if we configure the build correctly.

Would you guys be interested in a pull request for such a change?

Support a schema record. Each field in a schema would have the following properties:

• name (required) any object using the schema would have the same name
• JavaScript or plugin type (required) initially would support JavaScript object types 'string', 'int', 'date', 'boolean', but could be extended to support custom types (e.g. map object could include latitude and longitude)
• description (optional) primarily for documentation purposes
• validation (optional) must account for XSS

Schemas could be used for tabular or document based objects. Any object based on the schema would need to use the same names:

{ schema: { 
      A: { name: 'first-name', type: 'string', val: 'string at least 3 chars no more than 21 chars' } 
      A: { name: 'date-of-birth', type: 'date' } 
   },
   ASDF: { A: 'Bob', B: '345678098765' },
   FDSA: { A: 'Dan', B: '987654456789' }
}

The scheme does not need to be embedded in the metadata, it can be by reference.

Modify .blank based on other changes (#54) and rename to .not

Sadly the heroku server crashed from too much memory usage. So it looks like adding an LRU expiry or some type of flushing mechanism is going to be pretty important sooner rather than later.

Distributed Stress tests

Hi,

This project looks really neat, and i just have to ask a question. Should this presently be redubbed as a document-based database engine, instead of graph? I see the 'Ahead' that Neo4J cyphers plan to be added, but at the moment I don't see any aspects of a graph database - but enlighten me if I'm missing something, as that's very possible

Great job overall!

might be an issue with path() combos also, have a hacky fix that I'll push within a day if I can't get a good fix in.

While building the Four Consensus game...

The root node has a board field, which is either null or a sub object (a sub node).

When a game starts, the first move updates the board with a coordinate map.

We were listening to game.path('board').on(cb) and it SHOULD get called with either null or the board object.

Instead we occasionally get it called with the relation/soul of the new sub board object. This is incorrect. Needs to be fixed.

If we write an import feature from other databases to GUN we might want to prioritize something like Hadoop/Hive (or DAT?) or something like that. Because people are already probably using that and Hadoop has already built the drivers/import code for them for all other existing databases.

This way we can then use Hadoop to (practically speaking) import data from every where else.

Although I honestly don't think people should bother migrating into GUN. Why? Because data is the life blood of your app, so your entire app was probably built around the database, and converting/migrating is painful. GUN is forward facing for new apps, but people might be desperate enough from the hurt of their current system that they might want to do it anyways... and it would be nice to allow them to do so in a easier manner.

Currently it seems that this project is following some sort of compact style for the code base. The rules for it don't seem entirely clear. I would suggest adding some sort of style guide or a config for either js-beautify or ESLint so that it'd be easier for people to contribute and to keep a consistent style across all the code.

boohoo :(

hey i got your announcement email and the link to the web site gets a 400 bed request. just wanted to let you know dude :)

If there are many consecutive writes using the default test implementation, i.e. the file.js implementation, the disk is unable to handle the load and errors.

Currently, the only time file.js implementation should be used with gun is in local testing. In production usage, Amazon S3, Google Cloud Storage, localStorage, or an equivalent cold storage persistence layer should be used.

Note: This is a very low priority for the gun team. If this functionality is intriguing to you, please feel free to volunteer and start working on a modular implementation.

Support "table" objects in gun. These would be part of a SQL extension, but would be available for any tabular objects.

Each table has two meta objects:

• schema (see #89 )
• header
  • first - reference to first object's soul
  • last - reference to last object's soul
  • total - count of all objects ever in the table
  • active - count of active objects in the table

Each field in the table would need to be named according to the schema.

Each entry in the table has an additional boolean meta property of "deleted".

Example:

{ meta: {
    schema: { 
        A: { name: 'first-name', type: 'string', val: 'string at least 3 chars no more than 21 chars' } 
        A: { name: 'date-of-birth', type: 'date' } 
   },
    first: {'#': 'ASDF' },
    last: {'#': 'FADS' },
    total: 3,
    active: 2
   },
   ASDF: { A: 'Bob', B: '345678098765', deleted: 'false' },
   FDSA: { A: 'Dan', B: '987654456789', deleted: 'true' },
   FADS: { A: 'Amy', B: '765874890086', deleted: 'false' }
}

The scheme does not need to be embedded in the metadata, it can be by reference.

Quick question: could it work on the client-side with S3 only, without having any peers?

Explore Hadoop HDFS as a storage option (e.g. alternative to S3)

Adding Gun.Chain.map() would be really useful in moving toward some sort of "array support". It would also be handy if this method returns a promise that resolves when iteration is complete. I ran into a situation where I wanted to call $scope.$apply() after iteration was complete, but didn't know how to postpone that call until the .map() was complete.