I've implemented a search and replace feature and in this case it would be useful being able to apply the replace operations on the snapshot version at the time of the search; applying the replace operations on the current version could not work if a change was applied in the meantime.

It is currently possible to get a particular version of a document in read-only mode, but will it be possible to open it in read-write mode, such that any operation applied will be automatically merged with all the following versions?

This change follows on from discussion in this issue. Its primary aim is to allow clients to fetch an historical version of a document (a snapshot), by providing either:

• a version number, or
• a desired Date

The entry-point for this feature is added to the Connection class, deliberately separate from the Doc, because Doc is concerned with "live" document actions such as subscribing and submitting ops, whereas fetching an historical version of a document should not be associated with these ideas.

The feature is called with:

connection.getSnapshot(collection, id, version, callback): void;

The details of the interface are detailed in the README, and in the code documentation.

This change includes support for projections, and use of the readSnapshots middleware. It also hooks into Connection's hasPending method.

Performance optimisations are deemed out-of-scope for this change (see the issue for more details).

Note that this change also adds a development dependency on lolex which is used for mocking the time.

Introduction

I recently opened a pull request to add the ability to fetch snapshots of a document at a given version. This was first discussed in this other issue.

In that issue, the possibility of some optimisations was discussed briefly, but deemed out-of-scope for an initial offering. Now that it seems like that feature is close to being merged, I would like to start discussing how to optimise it further.

Motivation

These optimisations are far from just an academic exercise. We're currently running ShareDb on a production server, and we have some documents hitting 1,000,000+ operations.

Running the new getSnapshot method on a document like this takes on the order of minutes to return on a development machine, which is not acceptable. This sort of operation would ideally take < 10s independent of number of operations.

Deliverable

What I would like to establish with this issue ticket is a sensible optimisation to start work on.

Possible approaches

There are a number of possible approaches, some of which were discussed in the original issue. Those are recapped here.

Reversible types

The current approach for getSnapshot only works forwards, starting with the very first op and moving forwards. Depending on use case, it may often be likely that consumers are more interested in later versions of the document. In this case, starting from the current snapshot and working backwards would be much faster.

Pros

• Much faster to fetch recent versions of a document
• Roughly halves the worst-case time

Cons

• Making a type reversible is optional. For example, the [rich-text] OT type is not reversible. In these cases, this optimisation would not have any effect.
• Even when a type is reversible, the inversion can't always be trusted

Caching ops

A very large amount of time can be spent fetching ops. Anecdotally, it's responsible for over half the time spent retrieving a snapshot (using sharedb-mongo).

Subsequent version queries could be sped up if the ops are cached. Any requests for a version lower than the latest version in the cache can be fed directly from the cache.

However, potentially this sort of optimisation belongs at the driver level?

Pros

• Would be a huge speed boost over database drivers

Cons

• ‘There are only two hard things in Computer Science: cache invalidation and naming things.’ — Phil Karlton
• First fetch is still unaffected
• Could lead to vastly increased memory usage

Current version

Whenever we try to fetch the current snapshot through getSnapshot, it currently works its way through all the ops. We could instead trivially return the current snapshot. This is easy in the case of providing a null version, but would require a lookup of the current snapshot first in all other cases.

Pros

• Much faster for fetching current version

Cons

• Would require an extra snapshot lookup before every fetch, which is added overhead for an (unlikely?) scenario

Build from latest creation

When a document is deleted and then re-created, if we want a version after re-creation, we only need to fetch and apply the ops after that new creation.

However, as far as I'm aware, there's currently nothing defined in the driver API that lets us fetch the latest creation operation, which means currently we would still need to fetch all the ops, and then perform an initial pass through the ops to determine if such an optimisation could be made.

Pros

• Faster retrieval in some cases

Cons

• Possibly requires a change to driver API
• Otherwise still fetches all ops, and requires a second pass
• (Re-creation may be relatively uncommon?)

Snapshot milestones

In a similar vein of thinking to building from the latest creation op, if fetching and applying ops is expensive, one of the best things you can do is fetch and apply fewer of them.

We could achieve this by periodically saving snapshots of documents (every 100,000 ops, say). Then we could fetch the latest applicable snapshot milestone and reconstruct the rest of the snapshot on top of that.

Pros

• Limits the time of any retrieval
• Could be made configurable?

Cons

• Large infrastructure change
• Requires a change to the driver API
• Hard to add retroactively to existing documents

Conclusion

I think that the inversible types optimisation is probably the lowest-hanging fruit, but for purely selfish reasons this is the one I'm least likely to implement, because we use rich-text and our json0 ops fall into the trap I mentioned.

I also quite like the op caching idea, but the fact that first load is unaffected is quite a bummer.

I personally really like the idea of saving snapshot milestones. This is without doubt the optimisation that would be the most work, but I think that the ability to put a cap on the retrieval of any given document is absolutely invaluable. If people would be willing to support me on this change, I'd be more than keen to do this, because honestly I think we'll have to fork and implement this if I don't do it here.

Obviously, if there are any other optimisations I've missed, please pile in with comments.

It would be nice to modernize this code, this include :

• using es6 syntax (classes, exports, let, const)
• using promises and only promises
• using typescript or at least write a type definition
• dropping support for callback, browserchannel, etc.
• expect.js is a dead project
• add linter / formater
• callback with real Error objects, remove things like callback({ code: 4008, message: 'Unknown type' })

I tried to keep as close as possible to the original code. All tests are still passing but I haven't done any further testing.

• [x] Move all Lua scripts into lib/scripts as separate .lua files.
• [x] Convert to prototype-style.
• [x] Remove Makefile.
• [x] Make all current tests pass.
• [x] Implement query code tests
• [x] Update to 0.2.3

This new example uses React to encapsulate sharedb-string-binding as a component. It's bootstrapped from create-react-app. When complete, this will provide one example that addresses #109

Remaining things to do:

• [x] Use react-router to implement navigation between documents
• [x] Styles
• [x] Figure out how to leverage the create-react-app devtools AND connect to ShareDB

I'm submitting this incomplete PR now to solicit feedback on the approach. One thing I'm not totally certain about is memory/connection leakages - I'm assuming that invoking doc.destroy() is enough, but please let me know if I'm missing something.

When I restart my web socket server, the Connection on the client emits a state event with the newState set to "disconnected", which according to the documentation means:

'disconnected' Connection is closed, but it will reconnect automatically

However, the connection does not reconnect. Given that the socket which was passed to the Connection's constructor is now closed, I don't understand how reconnection would be possible. Don't we need a new WebSocket? What's going on here?

Hi,

Awesome work on this project.

Basically, I am trying to modify the users' request (for security reasons) on the server.

This means I do not have to explicitly share the direct DB information on the client.

I have tried using the middle ware to modify the receive request. However, as per the query middleware check below- it does not use the MODIFIED_REQUEST_NAME assigned (it just uses the original value).

What is the correct way to modify a request?

I have attempted the following:

 backend.use('receive', (request, done) => {
      request = {...request, data: {...request.data, c: MODIFIED_REQUEST_NAME } }
      done()
      // done(request) did not succeed
})

I also tried modifying it inside the query- no luck there either-

backend.use('query', (request, done) => {
          console.log('Query Request', request)
         request = { ...request, channel: MODIFIED_REQUEST_NAME, collection: MODIFIED_REQUEST_NAME, index: MODIFIED_REQUEST_NAME}
          done()
      })

Summary of changes

• This PR includes most commits from https://github.com/share/sharedb/pull/286 (hit a a dead end of red tests there)
• Split out presence logic completely from ShareDB core.
• The implementation of presence is exposed as statelessPresence, which can be used like this:

var ShareDB = require('sharedb');
var statelessPresence = require('sharedb/lib/presence/stateless');
var share = new ShareDB({ presence: statelessPresence })`).

• The value of the presence option is expected to contain implementations of the following classes:
  • DocPresence, which is instantiated at doc._docPresence.
  • ConnectionPresence, which is instantiated at doc._connectionPresence.
  • AgentPresence, which is instantiated at doc._agentPresence.
  • BackendPresence, which is instantiated at doc._backendPresence.
• Doc, Connection, Agent and Backend invoke methods on their respective presence classes.
• By default a dummyPresence is used, which implements methods that mostly no-ops.
• 100% of the presence logic is encapsulated within statelessPresence, so
• it should be possible to swap out better implementations in the future, and
• it's possible to develop third party presence implementations external to ShareDB.
• API compatibility is mostly retained between this and the Teamwork ShareDB fork.
  • The main difference is that the presence implementation needs to be passed into the constructor.
  • Presence is exposed as doc.presence, as in the original implementation.
• See this PR to compare these changes to the original Presence PR: https://github.com/Teamwork/sharedb/pull/3

This change adds the ability for clients to broadcast information about "Presence" - the notion of a client's position or state in a particular document. This might be represent a cursor in a text document, or a highlighted field in a more complex JSON document, or any other transient, current information about a client that shouldn't necessarily be stored in the document's chain of ops.

The main complication that this feature solves is the issue of keeping presence correctly associated with the version of a Doc it was created at. For example, in a "naive" implementation of presence, presence information can arrive ahead of or behind ops, which - in a text-based example - can cause the cursor to "jitter" around the change. Using the ShareDB implementation will ensure that the presence is correctly transformed against any ops, and will ensure that presence information is always consistent with the version of the document. We also locally transform existing presence, which should help to keep (static) remote presence correctly positioned, independent of latency.

In order to facilitate this, the feature must be used with an OT type that supports presence. The only requirement for supporting presence is the support of a transformPresence method:

type.transformPresence(presence, op, isOwnOperation): presence;

• presence Object: the presence data being transformed. The type will define this shape to be whatever is appropriate for the type.
• op Op: the operation against which to transform the presence
• isOwnOperation: boolean: whether the presence and the op have the same "owner". This information can be useful for some types to break ties when transforming a presence, for example as used in rich-text

This work is based on the work by @gkubisa and @curran, but with the following aims:

• avoid modifying the existing Doc class as much as possible, and instead use lifecycle hooks
• keep presence separate as its own conceptual entity
• break the presence subscriptions apart from Doc subscriptions (although in practice, the two are obviously tightly coupled)
• allow multiple presences on a single Doc on the same Connection

This goes along with https://github.com/share/livedb-mongo/pull/12. It handles emitting Missing operations errors when operations requested from the oplog are not there. This happens in submit and getOps (and by extension fetch, since it calls getOps).

This error should occur if a client has been offline longer than the ttl and tries to submit/fetch some ops. It should only occur if other edits have taken place while the user was offline, which have also expired. Specifically:

1. user 1 goes offline
2. user 1 edits
3. user 2 edits
4. ops are dropped (user 2’s ops expire)
5. user 1 comes back online and submits its ops
6. ERROR

If 3 and 4 are swapped (i.e. user 2 doesn’t edit anything before the ops are dropped), the error shouldn't occur. Basically, it should be fairly rare.

When the error happens, ShareJS sends it back to the client, which has an opportunity to notify the user that their changes could not be merged. Not ideal, but necessary to keep the data size in mongo from growing forever.

One case I thought of but I'm not sure about is whether we need to get the snapshot version in getOps when to is null to determine whether ops are expected from the oplog. Right now, the check for missing operations only occurs when there are ops returned (check ops[0].v === from). If to is null, we may also need to check if ops were expected if from < snapshot.v, which means fetching the snapshot there. I don't really like that, but it may be necessary. In submit this is handled already. What do you think?

At the moment, if you have many remote clients' presence in your Doc, the console logs a warning:

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 del listeners added. Use emitter.setMaxListeners() to increase limit
    at _addListener (webpack-internal:///./node_modules/events/events.js:211:15)
    at Doc.addListener (webpack-internal:///./node_modules/events/events.js:227:10)
    at RemoteDocPresence._registerWithDoc (webpack-internal:///./node_modules/sharedb/lib/client/presence/remote-doc-presence.js:45:13)
    at new RemoteDocPresence (webpack-internal:///./node_modules/sharedb/lib/client/presence/remote-doc-presence.js:21:8)
    at DocPresence._createRemotePresence (webpack-internal:///./node_modules/sharedb/lib/client/presence/doc-presence.js:25:10)
    at eval (webpack-internal:///./node_modules/sharedb/lib/client/presence/presence.js:115:21)
    at exports.digOrCreate (webpack-internal:///./node_modules/sharedb/lib/util.js:44:48)
    at Presence._receiveUpdate (webpack-internal:///./node_modules/sharedb/lib/client/presence/presence.js:114:29)
    at Connection._handlePresence (webpack-internal:///./node_modules/sharedb/lib/client/connection.js:822:26)
    at Connection.handleMessage (webpack-internal:///./node_modules/sharedb/lib/client/connection.js:249:19)

This is because we attach listeners for every remote client in the Doc.

Potential options to fix:

1. Do nothing — let consumers determine what a "sensible" number of max listeners is. In my opinon, this isn't a particularly nice option, and I think it should be the duty of the library to deal with warnings it's generating
2. Increment the max listeners ourselves — whenever we create a remote listener, we increment the number of allowed listeners by one (and decrement when destroying). This essentially excludes RemoteDocPresencefrom this listener count. The downside of this is that we also remove the *legitimate* warning case where consumers forget to destroy theirRemoteDocPresence`
3. Use some sort of global handler — have one event handler per doc for all remote presences. This is probably the most typical way to deal with this, although it does feel a bit like reinventing the wheel, and it potentially goes against the idea that the Doc should be unaware of the presence implementation. We also fall into the trap of solution 2 of not warning consumers about having lots of presence lurking in-memory (and at that point, maybe just implementing option 2 is simpler...?)
4. Other...?

This change adds a new request.$fixup() method to the SubmitRequest class, which allows middleware to mutate ops in the apply middleware.

Motivation

There are times when we want the server to make changes to ops submitted by a client. There may be many reasons for wanting to do this, including distrusting a client (eg setting timestamp, or auth information); keeping sensitive business logic out of client-side code; etc.

The only ways of doing this currently are:

1. Committing the client's op, and then submitting an amendment op from the server; or
2. Rejecting the client's op, and submitting the full amended op from the server

Option 1 works, but possibly commits badly-formed, or untrustworthy data to the database, and broadcasts it to remote clients. It also makes it possible to fetch historic snapshots of this untrustworthy in-between state, which undermines the trustworthiness of our data.

Option 2 also works, and might work with invertible types through a "soft" rollback, but can lose pending ops if we're not careful, or if the type isn't invertible.

Approach

This change adds a $fixup() helper function to the SubmitRequest object, named after the git action for performing similar actions.

The idea of the $fixup() function is to act as a middleware-friendly submitRequest().

Any ops submitted through $fixup() will be:

• composed with the request.op, so downstream middleware can work on the fixed-up op
• sent back to the original client, and applied
• help to avoid infinite middleware loops, since we don't trigger any nested submission calls

Note that in order to handle the case of a client disconnecting before getting their fixup, we now also store the fixup ops on the op m metadata field, and fetch this when getting ops to the current snapshot.

Limitations

This feature will only work:

• on types which implement the optional .compose() function
• in the apply middleware, where the snapshot's type is available, but before the op has actually been applied to the snapshot
• on creation or edit ops (not deletion)

Hi,

I’m trying to create a fully offline experience, not only storing the ops to send- but reading data from local storage (indexdb) when there’s no connection to the server.

My guess is something like:

1. for every item, to store the following in local storage:

{ data, src: connection.id, v: version, seq: connection.seq, c: collection, d: id, }

2. When unable to connect to the server, read items from local storage, recreating each one as a “doc” with normal properties (submitOp etc)

Is this plausible- can i simply recreate each “doc”?

Do you have any suggestion on implementation?

From time to time I receive this error on .submitOp. There are no specific actions. I can submit same action but change value from 1 to 2 fox example and will get this error. Also, it is not possible to reproduce, because the error can appear when i change 1 to 2, 2 to 3, n to m. You never know when this error appears.

Can someone clarify, in what cases this error appears? what exactly happening, so sharedb just says - no, not this time. And is there a way to workaround?

application not working during package implemention just have white screen in react app

when it import it doesn't work

Code;

import React, { useEffect, useState, useRef, useCallback } from "react"; import Quill from "quill"; import "quill/dist/quill.bubble.css"; import "quill/dist/quill.snow.css"; import Sharedb from "sharedb/lib/client"; import richText from "rich-text"; var Connection = require("sharedb/lib/client").Connection;

Sharedb?.types?.register(richText.type);

const RichTextarea = ({ id, collection, autoScroll }: any) => { const [socket, setSocket]: any = useState(null); const [connection, setConnection]: any = useState(null); const [quillObj, setQuillObj]: any = useState(null);

useEffect(() => { const _socket = new WebSocket("ws://wwww.localhost:3001"); const _connection = new Connection(_socket); setSocket(_socket); setConnection(_connection); }, []);

Package.json;

"dependencies": { "@apollo/client": "^3.5.10", "@chatscope/chat-ui-kit-react": "^1.9.7", "@chatscope/chat-ui-kit-styles": "^1.2.5", "@mantine/core": "^4.2.4", "@mantine/dates": "^4.2.4", "@mantine/form": "^4.2.8", "@mantine/hooks": "^4.2.4", "@mantine/notifications": "^4.2.4", "@mantine/rte": "^4.2.4", "@reduxjs/toolkit": "^1.8.1", "@stripe/react-stripe-js": "^1.10.0", "@stripe/stripe-js": "^1.35.0", "@types/react-credit-cards": "^0.8.1", "@types/react-helmet": "^6.1.5", "@types/sharedb": "^3.0.0", "@types/socket.io-client": "^3.0.0", "@types/stripe-v3": "^3.1.27", "@vime/core": "^5.3.1", "@vime/react": "^5.3.1", "axios": "^0.26.1", "browser-image-compression": "^2.0.0", "buffer": "^6.0.3", "dayjs": "^1.11.3", "firebase": "^9.6.11", "get-orientation": "^1.1.2", "graphql": "^16.3.0", "html-react-parser": "^1.4.10", "i18next": "^21.8.14", "i18next-browser-languagedetector": "^6.1.4", "i18next-http-backend": "^1.4.1", "moment-timezone": "^0.5.34", "quill": "^1.3.7", "react": "^17.0.0", "react-avatar-edit": "^1.2.0", "react-awesome-reveal": "^3.8.1", "react-credit-cards": "^0.8.3", "react-dom": "^17.0.0", "react-helmet": "^6.1.0", "react-hook-form": "^7.29.0", "react-i18next": "^11.18.1", "react-icons": "^4.3.1", "react-infinite-scroll-hook": "^4.0.3", "react-intersection-observer": "^8.34.0", "react-lottie-player": "^1.4.1", "react-markdown": "^8.0.3", "react-player": "^2.10.1", "react-redux": "^7.2.6", "react-router-dom": "^6.2.2", "react-video-play": "^2.0.3", "react-web-share": "^1.1.6", "rich-text": "^4.1.0", "sharedb": "^3.1.0", "sharedb-string-binding": "^1.0.0", "socket.io-client": "^2.3.1", "tabler-icons-react": "^1.53.0", "use-dynamic-refs": "^1.0.0", "web-vitals": "^2.1.0", "workbox-background-sync": "^6.5.3", "workbox-broadcast-update": "^6.5.3", "workbox-cacheable-response": "^6.5.3", "workbox-core": "^6.5.3", "workbox-expiration": "^6.5.3", "workbox-google-analytics": "^6.5.3", "workbox-navigation-preload": "^6.5.3", "workbox-precaching": "^6.5.3", "workbox-range-requests": "^6.5.3", "workbox-routing": "^6.5.3", "workbox-strategies": "^6.5.3", "workbox-streams": "^6.5.3" }, "devDependencies": { "@originjs/vite-plugin-commonjs": "^1.0.3", "@types/react": "^17.0.0", "@types/react-dom": "^17.0.0", "@types/react-lottie": "^1.2.6", "@types/react-redux": "^7.1.23", "@vitejs/plugin-react": "^1.0.0", "autoprefixer": "^10.4.0", "postcss": "^8.4.4", "sass": "^1.49.9", "typescript": "^4.3.2", "vite": "^2.6.4", "vite-plugin-pwa": "^0.12.0" }

I was researching about Edge Computing and it looks really cool. Just wanted to start a discussion - has anyone ever tried to run ShareDB in an Edge runtime like Deno or Cloudflare Workers? It seems doable in theory, but requires removing dependencies on many Node APIs.