Description

I installed typesense in my

ubuntu 20.04 apache server

and it is successfully running but I can't succeed in trying to sync my firestore data using the firebase extension. When I input the public IP of my server in the typesense host portion, it fails to upsert the data.

Steps to reproduce

firebase extension

and this is my config file for typesense server

I'm guessing that it needs SSL certification but where do I get my SSL certificates to put in the configs? Can self-signed work? Can certbot work?

I'm new to this so have mercy on me

""

Expected Behavior

Actual Behavior

Metadata

Typsense Version:

OS:

Description

Reading the docs i saw i have to create a new collection called typesense_sync with inside a document named backfill => trigger : true. But actually none of my documents are filled in the collection, in the logs i see few detail but i don't undestand

Steps to reproduce

Expected Behavior

Ther is a way to trigger backfill manually?

Actual Behavior

Metadata

Typsense Version:

OS:

Description

When creating or upserting a document, or a backfill operation the cloud functions fails with the code error "EAI_AGAIN getaddrinfo EAI_AGAIN https". The same issue occurs with both cloud functions (ext-firestore-typesense-search-backfillToTypesenseFromFirestore, ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite). The following are the logs from the cloud function:

• Function execution started
• Creating document {"id":"UMA-NICO-3"}
• Request #1629896880473: Request to Node 0 failed due to "EAI_AGAIN getaddrinfo EAI_AGAIN https"
• Request #1629896880473: Sleeping for 120s and then retrying request...
• Function execution took 120003 ms, finished with status: 'timeout' I tried running typesense on a virtual machine and typesense cloud, both cases with the same errors. I also tried creating a collection and a document from a JS client to the typesense cloud and worked fine. Firebase plan is BLAZE.

Steps to reproduce

Project created in Firebase, typesense cloud created, cloud function triggered by a new document (or update).

Expected Behavior

Documents created and updated on typesense cloud.

Actual Behavior

No documents or collections are synced from Fireabase to typesense cloud.

Metadata

Typsense Version: v0.21.0

OS: Linux Ubuntu (used on the VM)

Description

Hi, I am trying index some fields which are within an object that I am supposed to flatten in order to index in Typesense. But looking at this, https://github.com/typesense/firestore-typesense-search/blob/master/functions/src/utils.js#L16-L20 , I think even a flattened field would not match. Is there a workaround for fields within an object using firestore & typesense? Thanks in advance.

Steps to reproduce

Place a flattened field into Firestore Collection Fields.

Expected Behavior

Index a flattened field within an object into typesense.

Actual Behavior

Unable to index a flattened field within an object into typesense.

Metadata

Typsense Version: typesense/[email protected]

Hi all! i've create a typesense schema like this: const myCollection = { 'name': 'products', 'fields': [ {"name": ".*", "type": "auto" } ] } then, i've installed firestore extension and link "products" schema to a firestore DB collection. But the functions doesn't work.

When I tried backfill, function throw this error:

When I tried to add a document the function doesn't throw an error but in the typesense schema doesn't appear. On delete the same document, the function throw this error:

It seems that firestore extension cannot comunicate with typesense schema. Can you help me?

Description

Hi! I want to implement full text search into my project which is based on Firebase. Right now I'm trying to get some things rolling by backfilling my data to the TypeSense Collection. I'm using a Firebase TypeSense Extension from this repo. Everything went smoothly except the backfilling process. The ext-firestore-typesense-search-backfillToTypesenseFromFirestore cloud function gives me an error "Memory limit exceeded". Usually I fix that problem by deploying a function once again with higher available resources but I'm not sure if this is the proper solution. The test collection has a size of around 60MB and contains 53K documents. I was able to successfully export other collection which is a bit smaller (around 30k documents).

Steps to reproduce

Expected Behavior

The ext-firestore-typesense-search-backfillToTypesenseFromFirestore trigger function shouldn't crash.

Actual Behavior

If the collection is big enough the ext-firestore-typesense-search-backfillToTypesenseFromFirestore trigger function returns an error Function invocation was interrupted. Error: memory limit exceeded.

Metadata

Typsense Version: 0.21.0

Change Summary

Currently, if there is a GeoPoint value in your document you will receive this error log printed:

• convert geo-point objects to a 2 element array
• convert timestamp dates to Unix timestamps

PR Checklist

• [x] I have read and signed the Contributor License Agreement.

Description

This is not a bug with the extension, but a request to add a feature that would allow support for multiple collections to be indexed into typesense. The configuration steps could allow an array or list of collections to be inputted, and a subarray for fields.

Hi, I've been trying to setup this for a while now and I never get any errors but nothing happens when I update or add docs to the collection it was supposed to be indexing. How to troubleshoot the change in the document? What commands should I use?

The only log I can see is when running tail -f /var/log/typesense/typesense.log

I20220618 02:35:29.949776 28495 raft_server.h:60] Peer refresh succeeded!
I20220618 02:35:39.951146 28490 raft_server.cpp:531] Term: 5, last_index index: 18, committed_index: 18, known_applied_index: 18, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 44

Cloud logs return 0 logs.

My setup:

• I have a fresh Ubuntu AWS Lightsail running typesense
• port 8108 is open
• https is on
• health check returns ok.
• Collection was created on the server
• typesense_sync collection created

Questions:

• Can we please expand the readme to include how to troubleshoot?
• Can we please add to the readme examples of the config? I.e. do I need a forward slash for collections? Do I need the port on the server?

I am using this to test if a change in a doc creates any new document but It always return 0 documents.

curl -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
     -X GET \
    "https://EXAMPLE.com:8108/collections/service"
{"created_at":1655499258,"default_sorting_field":"","fields":[{"facet":false,"index":true,"infix":false,"locale":"","name":".*","optional":true,"sort":false,"type":"auto"}],"name":"service","num_documents":0,"symbols_to_index":[],"token_separators":[]}%       

hi all. I'm running self hosted typesense on AWS via docker. I have nginx redirecting 80 and 443 traffic to locahost:8108, with SSL certificate etc.

if I visit my-domain.com/health, I get {"ok":true} back. I believe this means nginx is correctly redirecting traffic. however, when I try to trigger calls on firestore, I get this error:

{
  "textPayload": "Error: Request failed with HTTP code 404 | Server said: Not Found\n    at ApiCall._customErrorForResponse (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:378:26)\n    at ApiCall._callee$ (/workspace/node_modules/typesense/lib/Typesense/ApiCall.js:186:70)\n    at tryCatch (/workspace/node_modules/regenerator-runtime/runtime.js:63:40)\n    at Generator.invoke [as _invoke] (/workspace/node_modules/regenerator-runtime/runtime.js:293:22)\n    at Generator.next (/workspace/node_modules/regenerator-runtime/runtime.js:118:21)\n    at asyncGeneratorStep (/workspace/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)\n    at _next (/workspace/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
  "insertId": "000000-1be6577f-97bb-4c82-aa28-74f47012e713",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "modify-dev",
      "region": "us-central1",
      "function_name": "ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite"
    }
  },
  "timestamp": "2021-10-21T21:37:53.973Z",
  "severity": "ERROR",
  "labels": {
    "execution_id": "k67afrdv37p2"
  },
  "logName": "projects/modify-dev/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/modify-dev/traces/0f171f3a97592d67f6c842091303f0b5",
  "receiveTimestamp": "2021-10-21T21:38:03.542328289Z"
}

when looking at my nginx access logs, I can see that the extension does hit the server. any ideas?

Description

I have setup the typesense extension in my Firebase project, but when trying to add a document, i get the following error:

Error detected in ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.Insight","errorGroup":"CNTlnprCi-zYJw","exceptionInfo":{"type":"Error","message":"timeout of 10000ms exceeded"},"errorEvent":{"eventTime":"2021-07-08T20:47:32.503Z","message":"Error: timeout of 10000ms exceeded\n at createError (/workspace/node_modules/axios/lib/core/createError.js:16:15)\n at RedirectableRequest.handleRequestTimeout (/workspace/node_modules/axios/lib/adapters/http.js:280:16)\n at RedirectableRequest.emit (events.js:376:20)\n at RedirectableRequest.emit (domain.js:532:15)\n at Timeout._onTimeout (/workspace/node_modules/follow-redirects/index.js:166:12)\n at listOnTimeout (internal/timers.js:555:17)\n at processTimers (internal/timers.js:498:7)","serviceContext":{"resourceType":"cloud_function","service":"ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite"}}}

I am hosting the server on my local machine for the time being, so perhaps the problem is stemming from the host name that I used when setting up the extension. I set up port forwarding in order to get a global IP address, and used that address XX.XX.XX:8108 as the host name. Sending curl requests via the terminal to that host name works, however.

Change Summary

If Typesense self hosted cluster is not running at 443 port, firestore will not sync changes

PR Checklist

• [x] I have read and signed the Contributor License Agreement.

Hi Typesense Team

i get the following message when i try to install this extension with the console and cli

RESOURCE_ERROR at /deployments/firebase-ext-firestore-typesense-search/resources/backfillToTypesenseFromFirestore: {"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":"Build failed: {"metrics":{},"error":{"buildpackId":"","buildpackVersion":"","errorType":"OK","canonicalCode":"OK","errorId":"","errorMessage":""},"stats":[{"buildpackId":"google.nodejs.runtime","buildpackVersion":"1.0.0","totalDurationMs":1496,"userDurationMs":0},{"buildpackId":"google.utils.archive-source","buildpackVersion":"0.0.1","totalDurationMs":45,"userDurationMs":44},{"buildpackId":"google.nodejs.npm","buildpackVersion":"1.0.0","totalDurationMs":6428,"userDurationMs":6185},{"buildpackId":"google.nodejs.functions-framework","buildpackVersion":"0.9.4","totalDurationMs":3756,"userDurationMs":3541},{"buildpackId":"google.config.entrypoint","buildpackVersion":"0.9.0","totalDurationMs":0,"userDurationMs":0},{"buildpackId":"google.utils.label","buildpackVersion":"0.0.2","totalDurationMs":0,"userDurationMs":0}],"warnings":null,"customImage":false}"}; RESOURCE_ERROR at /deployments/firebase-ext-firestore-typesense-search/resources/indexToTypesenseOnFirestoreWrite: {"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":"Build failed: {"metrics":{},"error":{"buildpackId":"","buildpackVersion":"","errorType":"OK","canonicalCode":"OK","errorId":"","errorMessage":""},"stats":[{"buildpackId":"google.nodejs.runtime","buildpackVersion":"1.0.0","totalDurationMs":1485,"userDurationMs":0},{"buildpackId":"google.utils.archive-source","buildpackVersion":"0.0.1","totalDurationMs":59,"userDurationMs":59},{"buildpackId":"google.nodejs.npm","buildpackVersion":"1.0.0","totalDurationMs":6677,"userDurationMs":6425},{"buildpackId":"google.nodejs.functions-framework","buildpackVersion":"0.9.4","totalDurationMs":3689,"userDurationMs":3477},{"buildpackId":"google.config.entrypoint","buildpackVersion":"0.9.0","totalDurationMs":0,"userDurationMs":0},{"buildpackId":"google.utils.label","buildpackVersion":"0.0.2","totalDurationMs":0,"userDurationMs":0}],"warnings":null,"customImage":false}"}

when i look at the cloud functions there is a warning that the functions could not be built - log message from the function:

Build failed: {"metrics":{},"error":{"buildpackId":"","buildpackVersion":"","errorType":"OK","canonicalCode":"OK","errorId":"","errorMessage":""},"stats":[{"buildpackId":"google.nodejs.runtime","buildpackVersion":"1.0.0","totalDurationMs":1437,"userDurationMs":0},{"buildpackId":"google.utils.archive-source","buildpackVersion":"0.0.1","totalDurationMs":42,"userDurationMs":42},{"buildpackId":"google.nodejs.npm","buildpackVersion":"1.0.0","totalDurationMs":6883,"userDurationMs":6640},{"buildpackId":"google.nodejs.functions-framework","buildpackVersion":"0.9.4","totalDurationMs":3529,"userDurationMs":3323},{"buildpackId":"google.config.entrypoint","buildpackVersion":"0.9.0","totalDurationMs":0,"userDurationMs":0},{"buildpackId":"google.utils.label","buildpackVersion":"0.0.2","totalDurationMs":0,"userDurationMs":0}],"warnings":null,"customImage":false}

Thanks Michael

Description

As noted in the Firebase functions documentation, background function triggers are not guaranteed to be received in the order the changes occured.

When rapid changes occur in the database, out-of-order events will result in out-of-order processing, meaning Typesense can end up with stale data in it which is not consistent with the final Firestore resting state after the influx of changes.

From all of the reading I've done, it is a fools-errand to try and fight the system to process them in order -- nevertheless without a solution, extensions like this are difficult to use in production environments.

Steps to reproduce

1. Write a demo script / perform an app action rapidly that writes to the database
2. Observe that the events are received out of order
3. Quite often, this will result in Firestore containing the wrong data from an old event

Expected Behavior

Regardless of the order the events are received, only events newer than the last-processed event should be processed.

Actual Behavior

Inconsistent data in Typesense

Other Findings / Info

In trying to solve for this, I've attempted to use a Firestore collection to "lock" the item being updated, send to Typesense, and then unlock for future events after writing event context back to the lock endpoint.

Simplified example (RTDB trigger used here, but Firestore is affected by this as well):

.database.ref(`tasks/{itemId}`)
.onWrite(async (change, context) => {
  /**
   * We use a Firestore transaction to create a "lock" at a DB location
   * for a given `itemId`
   */
    const { timestamp, eventId } = context;
    const { itemId } = context.params;
    const timestampRef = firestore
      .collection(`typesenseLocks_tasks`)
      .doc(itemId);
    await admin.firestore().runTransaction(async transaction => {
      const dataChangedTimestamp = new Date(timestamp).getTime();
      const lastUpdatedTimestampDoc = await transaction.get(timestampRef);
      const lastUpdatedData = lastUpdatedTimestampDoc.data();

      /**
       * If this is the first time this document was changed (no previous locks),
       * or the last-stored lock timestamp is older than the current event's timestamp,
       * prepare a payload and send to Typesense.
       */
      if (
        (!lastUpdatedData?.timestamp ||
          dataChangedTimestamp > lastUpdatedData.timestamp) &&
        lastUpdatedData?.eventId !== eventId
      ) {
        // Send to Typesense
        await updateTypesense(change, indexer, itemId);

        // Finalize Transaction by writing the event timestamp and eventId back to Firestore for comparison by the next events
        transaction.set(timestampRef, {
          timestamp: dataChangedTimestamp,
          eventId
        });
      } else {
        /**
         * Do nothing, current event is older than last-indexed event already recorded, can be safely discarded
         */
      }
    });

In an attempt to fix this, I've tried moving the await updateTypesense(change, indexer, itemId); line inside and outside of the transaction function block, but in both scenarios stale data ends up inside Typesense.

From logs I've added, the above transaction appears to work, old events are dropped if a newer one has already been processed. But if two events that are extremely close together are processed in the correct order, Typesense occasionally ends up with the first event's payload.

This led me to believe that perhaps Typesense was parallelizing indexing operations, but since they are confirmed as serialized, I'm back at square one.

Metadata

Typsense Version: v0.23 OS: Typesense Cloud Relevant Slack Thread: https://typesense-community.slack.com/archives/C01P749MET0/p1655047418496499

Description

I am using the Firebase CLI to try to enable the extension for a Firebase project. Major note is that the last few times I did this it worked fine, so this seems to be a new issue. I think it may be related to this change: https://github.com/typesense/firestore-typesense-search/commit/44c5b37ce84ea2573cc40419f0e414865bbc4544

The TYPESENSE_API_KEY parameter appears to be the issue. Previously I had this set as the admin API key, which used to work. Now, when I try to enable the extension, I get an error related to secrets manager, which seems to be saying I have formatted it wrong. Specifically it says it should be formatted like projects/*/secrets/*/versions/* instead of just the API key, but I am not sure exactly how to format my API key like that (I am not sure which each of the asterisks in this case should be)

Steps to reproduce

Set up parameters file Run Firebase command to enable extension

firebase ext:install typesense/firestore-typesense-search --project=projectId --params=params.env --force

Expected Behavior

The extension will enabled for my Firebase project

Actual Behavior

I receive a HTTP 400 error:

Error: HTTP Error: 400, Secret ID 123.... is invalid, must be of form projects/*/secrets/*/versions/*

Metadata

Typesense Version: v0.23.0.rc27

OS: MacOS

Description

This is a request to support backfill all subcollections return from a collection group query. This could be helpful in use case like, backfill landmarks data from all cities under path: cities/{cityId}/landmarks.