Context

This is an alternative to #20.

Problem

There is a race condition between the processing of messages in start() and the list() API. list() is supposed to have all the up-to-date information, but by just querying ssb-box2 (and ssb-keyring), it doesn't, because there may be a message from the live query in start() that still hasn't trickled down and made its way to ssb-box2 (and ssb-keyring).

Solution

We figured (pair programming with @Powersource) that we don't need to strictly solve the race condition in list() if we embrace eventual consistency in the apps, but in tests, we need to make strictly solve them, to make the correct assertions.

This PR adds onCaughtUp(cb) as an internal API just for our tests. The way it works is a bit hacky, I wanted to use ssb.db.prepare() (which in turn is just a light wrapper for jitdb's prepare()) but the problem is that prepare() only syncs the bitvector indexes, it doesn't fetch the messages from the log (which is an asynchronous operation with the file system). That meant that the processing in start() is "slower" than jitdb prepare() which would meant that we are not yet caught up.

That's why I added the comment:

// This is a bit hacky, because it fixes the race condition with `start()`
// by relying on ssb-db2 internals. This only works because the query in
// `start()` was initiated before this one, and because ssb-db2 and jitdb
// have an internal queue of queries that are executed in order.

@arj03 Since you're good at race conditions, and since this is a hacky solution, I'd really appreciate your thoughts on this PR.

Fixes https://github.com/ssbc/ssb-tribes2/issues/13

• [x] Maybe do https://github.com/ssbc/ssb-uri2/issues/18 before merging to make the code here cleaner
  • [x] If we do the above then uninstall ssb-ref again
• [x] Should we update buildGroupId to not calculate on sigil keys? I guess? no. see comment below
  • [x] We might need to update the envelope spec? Depends on how encode works i guess? What's SLP? https://github.com/ssbc/envelope-spec/tree/master/cloaked_msg_id

It's the future :sparkles:

Made some related issues https://github.com/ssbc/private-group-spec/issues/17 https://github.com/ssbc/ssb-uri-spec/issues/15

Spiritual successor to the meta-feeds branch.

Implements the metafeeds group spec.

Asking for a review from @Powersource.

And for visibility, pinging @mixmix .

Based on https://github.com/ssbc/ssb-tribes2/pull/8

Fixes https://github.com/ssbc/ssb-tribes2/issues/10

Doing this for now since I think it'll cover our usecases, but in the future I guess we should do https://github.com/ssbc/ssb-tribes2/issues/9

based on https://github.com/ssbc/ssb-tribes2/pull/2

We need this to add correct group tangles to group messages

• [x] probably combine addGroupTangle and create into a function with better dx, that we then probably expose in the top level api

As an alternative to calculating the size https://github.com/ssbc/ssb-tribes2/pull/8

• [x] Import needed code from other PR
• [x] Remove unused code
• [x] Add test with 5000 msgs and time it

In the current tanglePrune code from tribes1 it is able to prune until the previous array is empty. Is that a good idea or likely to break tangle logic?

https://github.com/ssbc/ssb-tribes2/blob/ff6905b7638220073f8c292cee648d8e99cc8a49/lib/tangle-prune.js#L23-L28

related https://github.com/ssbc/ssb-tribes2/issues/9

right now i'm adapting this pr https://github.com/ssbc/ssb-tribes2/pull/8 to assume we're publishing on buttwoo instead of classic (like old tribes1). i've swapped json.stringify for bipf size checking but the size is still limited to 8k. buttwoo supposedly has a limit of 16k which changes things.

Working on https://github.com/ssbc/ssb-tribes2/pull/8 there's code in tanglePrune that assumed how the content of a msg is stringified. In tribes1 it assumed classic with json.stringify, now I'm thinking of assuming buttwoo with bipf. But it should probably be generalized or at least documented that it (or at least that feature) only supports 1 format.

Fixes https://github.com/ssbc/ssb-tribes2/issues/18

• [x] add for create
• [x] add for addMembers
• [x] include tests from tribes1
• [x] see that tests actually pass once https://github.com/ssbc/ssb-tribes2/pull/20 is merged
• [x] wait for https://github.com/ssbc/private-group-spec/pull/19 to be merged and published and update package.json here

It's disabled for some reason

https://github.com/ssbc/ssb-tribes2/blob/206138ecabb88f0f807e4d3e2218808d53db79ab/index.js#L21

https://github.com/ssbc/ssb-tribes2/blob/206138ecabb88f0f807e4d3e2218808d53db79ab/index.js#L196-L198

The addMember API should be renamed to "inviteMember".

The start() listener should NOT add the keys to ssb-box2, should NOT perform reindexEncrypted.

A new API listInvites that gives a pull-stream source of all the invites (group/add-member) you have received but haven't yet "accepted".

A new inviteMember API that allows you to acknowledge a group/add-member, add the keys to ssb-box2, and begin reindexEncrypted.

We have this issue for making the member tangle work properly https://github.com/ssbc/ssb-tribes2/issues/28 which i imagine means setting the previous correctly. But until we do that we shouldn't break stuff for ourselves in the future. Currently when a member is added we make the member tangle point at the group root msg.

https://github.com/ssbc/ssb-tribes2/blob/133fcefe9a2c910c0226f2b284f9c6688518a805/index.js#L135-L138

But that doesn't really make sense since in that root message we don't init the member tangle

https://github.com/ssbc/ssb-tribes2/blob/133fcefe9a2c910c0226f2b284f9c6688518a805/index.js#L49-L54

Easiest way to fix this is to add

members: { root: null, previous: null }

after line 52. It would maybe make more sense to only init it once we need it (on first member add i.e. admin adding themselves) but that's slightly more complicated.