From a discussion with @cpursley: "Ultimately the end user might want to use that change data to update other records in their app. Perhaps a background job that queries nango tables for changes?"

As pointed out by @cpursley, webhooks are backend agnostic (compared to callbacks) and could be the solution to avoid periodically polling Nango-synced data to update other records.

Currently when Nango does a refresh it fetches all the records from the remote, checks if they exist in the DB (using the unique_key) and depending on that updates the existing record or adds a new one.

This means that if a record has been deleted on the remote, and hence no longer gets returned in the response, we don't do anything and just keep the old record around. This is one possible way to handle this scenario, another would be to also delete the record in the DB. A third option would be to mark it as deleted first and then remove it in a subsequent run (this would give the application a chance to act on the record before it gets removed).

We are not sure yet which behaviour is best, if you have a use case for this please leave a comment here so we can gather them.

We current store all Nango-related data in the default public schema of Postgres. We should use a separate schema for clarity and make it configurable.

Temporal.io is an OSS library that helps with handling scheduling, tasks and workers. This article shows how Airbyte leverages it and the types of benefits it provides.

I am already starting to re-implement some basic features about scheduling (e.g. cron), jobs (e.g. status, retries), etc. that would be better handled by Temporal.

Temporal also plays well with self-hosting and Postgres that we already use.

I am running the Nango worker in a container and connecting to my app on the host via host.docker.internal.

The host app runs using self-signed SSL certs when debugging with Visual Studio. This is causing the webhook to fail with this message:

[ERROR] Webhook response error from https://host.docker.internal:5001/webhooks/nango for job 3 Sync 2: self-signed certificate (check debug logs for details).

To validate with curl, the following fails with the same error:

curl -i -X POST -H "content-type: application/json"  --data '{ "type":"HELLO" }' https://host.docker.internal:5001/webhooks/nango

But, if I add the --insecure flag it works.

One suggestion would be to add an "IGNORE_SSL" flag to the env vars and handle the same way to curl?

In the docs under "https://docs.nango.dev/nango-sync/schema-mappings" it reads:

"If you only specify a table name in the mapped_table parameter, e.g. pokemon, Nango will use (or create) the table in the default schema of your Postgres database (by default this is called public)."

This appears to work the opposite way - if I don't specify a schema the table gets created under the nango schema. If I specify "public.tablename" it get's created in public schema.

I think I prefer the "nango as default schema" approach so consider correcting the docs.

From the #help-and-questions channel of the Slack community:

npm run start syncRedditSubredditPosts redditapi throws errors and db table is not filled.

Logs: nango_sync_error.log

The problem seems to be that the DB string types we’re using are too small for some of the fields returned by the Reddit API.

For simplicity sake, we are starting with a default sync frequency of 1h. It should be easy to (configure this sync frequency via:

• The sync call (using the config object?)
• Later on with a CLI and dashboard

The Shopify Admin REST API uses cursors for pagination, but these are sent to the client as part of the response headers (instead of the JSON body): https://shopify.dev/api/usage/pagination-rest

A real world example is the orders endpoint: https://shopify.dev/api/admin-rest/2022-10/resources/order#get-orders?since-id=123

Objective

When an HTTP call to the API fails due to a rate-limit error we want Nango to detect this and automatically implement an appropriate backoff strategy.

Tasks

• [ ] Detect rate-limit issues by looking at response status code & headers
• [ ] Implement backoff (with X-After header if available)
• [ ] Add configuration per Integration to parametrize rate-limit detection & mitigation tactics
• [ ] Add proactive throttling if potential rate-limit issue is forecast (so we don't run into it)
• [ ] Make sure access token refresh is retried if it fails due to a rate-limit issue

Maybe out of scope for a first version but worth thinking about: Some rate-limits are for very long periods of time, eg. daily limit is exceeded. How should we handle these? If we just wait the main application may wait a veeery long time for triggerAction to complete (or run into issues itself if it is e.g. called from an AWS lambda which have ~15min timeout)

Currently when Nango runs in the docker container the log files get written to a file that is in the docker container. This makes it hard for users to access these logs and debug issues.

When the docker containers are run with docker compose (and we thus know the user is running it from within the Nango repo) we should write these doc files to logs/combined.log and logs/error.log in the repository so the user can easily inspect them.

One solution for this could be to use bind volumes, which would easily allow us to bind the logs folder into a folder in the container. The old Nango has an example of this here: https://github.com/NangoHQ/nango/blob/7dfa5ffb02b98af13bc608a04621daf73f6bb13e/docker-compose.yaml This will also help with production deployments as it gives users a way to specify where log files should get written.

Currently, the integration tests only test the 1st job of each Sync. This will no catch problems due to updating & deleting fields. We should run the tests so that each Sync runs multiple jobs.

We recently added a bunch of integration tests. They need to be triggered manually at this point.

We should integrate this to our CI. The main blocker I see is that these tests hit real APIs (e.g. Hubspot) which have already returned some errors due to rate limiting. An easy workaround would be to skip the test when we get a rate limit reponse.

The Sync config is basically a JSON blob.

To simplify examples in the docs (and codebase), as well as reduce docs updating work, we could centralise each Sync's configuration for all languages in a single JSON file.

With Nango v0.4.0 I ran the syncGithubStargazers example:

npm run example syncGithubStargazers NangoHQ nango 1

The first job runs fine but when I inspect the _nango_jobs table afterwards it shows that 201 records are unchanged. This is not possible as this is the first run of this Sync ever, correct are that 201 records have been added.

There is a retry policy with back-off between jobs, but not within one job that fetches many pages.

The default fix for hitting a rate limit when fetching many pages is to adjust the page size + sync frequency.

As an additional measure, we could keep the job alive and let it wait to fetch the remaining pages.