Description

New GitHub dashboard. Will be used as outcome of our Quickstart.

TODO:

• [x] Date Filter - difficult on the PR cycle time SQL question
• [x] Repository Filter - difficult because tasks board id and prs repository id do not have the exact same values
• [x] Link in Welcome dashboard
• [x] Upgrade metabase dashboard import to support SQL filters

Type of change

• New canned dashboard

Checklist

(Delete what does not apply)

• [x] Have you checked to there aren't other open Pull Requests for the same update/change?
• [x] Have you added an explanation of what your changes do and why you'd like us to include them?

Description

Follow up on #169 and depends on https://github.com/faros-ai/faros-events-cli/pull/77

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

• [x] Have you checked to there aren't other open Pull Requests for the same update/change?
• [x] Have you lint your code locally before submission?
• [x] Have you added an explanation of what your changes do and why you'd like us to include them?
• [x] Have you written new tests for your core changes, as applicable?
• [x] Have you successfully run tests with your changes locally?

Description

Adding authorization header for Hasura restful api call for allowing the mock data script to successfully upload/delete data form the api. The header value is being loaded from the project's main .env file which contains the necessary environment variables for running the entire project, including Hasura's admin secret.

Fixes #91

Type of change

(Delete what does not apply)

• Bug fix (non-breaking change which fixes an issue)

Checklist

(Delete what does not apply)

• [x] Have you checked to there aren't other open Pull Requests for the same update/change?
• [x] Have you lint your code locally before submission?
• [x] Have you added an explanation of what your changes do and why you'd like us to include them?

OBS: There are no tests written for the mock-data package

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

(Delete what does not apply)

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

(Delete what does not apply)

• [ ] Have you checked to there aren't other open Pull Requests for the same update/change?
• [ ] Have you lint your code locally before submission?
• [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
• [ ] Have you written new tests for your core changes, as applicable?
• [ ] Have you successfully run tests with your changes locally?

Description

On windows, one sees local volume related errors (see #173 and this slack thread). However, we noticed that:

1. on that same machine, one could start vanilla airbyte with docker compose (see airbyte documentation)
2. on that same machine, docker-compose -f airbyte-services.yaml up did not have any volume issues (i.e. airbyte-server got successfully created.

Hence, there may be some issues with the extends keyword, which is not surprising given its checkered history (here, and here).

The fix is to group everything into a single compose file. Fixes #173

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

(Delete what does not apply)

• [x] Have you checked to there aren't other open Pull Requests for the same update/change?
• [x] Have you added an explanation of what your changes do and why you'd like us to include them?
• [x] Have you successfully run tests with your changes locally?

Description

Upgrades to Airbyte 39.

Type of change

• New feature (non-breaking change which adds functionality)

Checklist

(Delete what does not apply)

• [x] Have you checked to there aren't other open Pull Requests for the same update/change?
• [x] Have you added an explanation of what your changes do and why you'd like us to include them?
• [x] Have you successfully run tests with your changes locally?

Description

All supported sources now have a canned source and a corresponding connection to the Faros destination in Airbyte.

Related to #195

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Checklist

• [x] Have you checked to there aren't other open Pull Requests for the same update/change?
• [x] Have you added an explanation of what your changes do and why you'd like us to include them?
• [x] Have you successfully run tests with your changes locally?

Description

This PR adds initial support for Kubernetes deployments through Kustomize. This allows for a more K8s flexible deployment K8s model that can cater to users particular requirements.

It uses Airbyte's provided Kustomize distribution, and include the additional dependent resources (metabase, n8n, hasura).

There are still improvements that can be made for a true production deployment, but I think it's a good starting point to get feedback and have it integrated.

Currently it contains a base setup with no overlays, that mimics much like the local development setup provided through docker-compose. As we move forward we can start creating overlays to cater to general use cases.

Noticeable features:

• Auto Initialization using faros-init docker image (farosai/faros-ce-init:latest)
  • This is done by using a K8s Job that is deployed along with the stack
• Creation of ConfigMap based on basic .env for auto setup
  • For this I ended up creating a hard link from the root file so that we could run kustomize without the flag --load-restrictor LoadRestrictionsNone. This flag prevents files from being loaded outside of the kustomization.yaml folder.
  • This provides a ConfigMap that can be mounted onto the appropriate manifests
• Easy overrides with Kustomize which solves for secret management, security requirements, and other governance needs

Deployment

For deploying this stack provided that you have access to a cluster you can just run:

kubectl apply -k kube/base

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Checklist

(Delete what does not apply)

• [x] Have you checked to there aren't other open Pull Requests for the same update/change?
• [x] Have you lint your code locally before submission?
• [x] Have you added an explanation of what your changes do and why you'd like us to include them?
• [x] Have you written new tests for your core changes, as applicable?
• [x] Have you successfully run tests with your changes locally?

Add scripts to upload and delete mock data for DORA dashboard.

Ensure you are running the latest version of Faros CE.

From root directory

docker-compose stop; ./start.sh

Then get into the mock-data directory and install dependencies

cd mock-data && npm i

Usage

Upload data

./bin/mock-data upload

Deleting data uploaded by script

./bin/mock-data delete

Use -u flag to specify Hasura service url if using a different endpoint

Describe the bug Related to the Faros performant fork of GitHub connector that was made default in (#140) and released as part of v0.2.0.

The Faros fork of the Airbyte Github Source does not appear to be built for M1 Mac architectures (ie. linux/arm64). I looked around to see where I might make a PR to update it but I cannot find the source of that fork. Additionally, the same error occurs for the faros fork of the airbyte-faros-destination connector.

To Reproduce Steps to reproduce the behavior:

1. Follow the QuickStart guide
2. In Step 2, configure the Github source
3. Run a sync and check the logs
4. Relevant log pasted below
5. Operative errors are the logline below which occurs for both farosai/airbyte-faros-destination and farosai/airbyte-github-source connectors Error: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Expected behavior A full sync of the configured Github repos

Screenshots and logs

2022-09-25 16:11:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.access_token: is missing but it is required, $.option_title: must be a constant value OAuth Credentials
2022-09-25 16:11:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 INFO i.a.a.LoggingTrackingClient(track):43 - track. version: 0.39.37-alpha, userId: 00000000-1111-2222-3333-444444444444, action: Connector Jobs, metadata: {job_type=sync, config.source.credentials.option_title=PAT Credentials, config.destination.source_specific_configs.bitbucket.application_mapping=set, namespace_definition=source, config.destination.edition_configs.segment_user_id=set, config.source.credentials.personal_access_token=set, config.destination.dry_run=false, frequency=manual, connector_source_definition_id=00000000-1111-2222-3333-444444444445, workspace_id=00000000-1111-2222-3333-444444444446, config.destination.source_specific_configs.jira.use_board_ownership=true, attempt_stage=STARTED, config.destination.edition_configs.hasura_url=set, config.destination.source_specific_configs.squadcast.application_mapping=set, attempt_id=1, connector_destination=Faros Destination, catalog.sync_mode.incremental=set, connector_destination_docker_repository=farosai/airbyte-faros-destination, table_prefix=true, config.destination.jsonata_mode=FALLBACK, workspace_name=00000000-1111-2222-3333-444444444446, config.source.start_date=set, connector_source=GitHub (fork by Faros AI), config.destination.source_specific_configs.jira.exclude_fields=set, connector_source_docker_repository=farosai/airbyte-github-source, config.destination.source_specific_configs.pagerduty.application_mapping=set, config.destination.source_specific_configs.victorops.application_mapping=set, config.source.repository=set, config.destination.invalid_record_strategy=SKIP, catalog.sync_mode.full_refresh=set, connection_id=00000000-1111-2222-3333-444444444447, config.destination.edition_configs.edition=community, job_id=1, connector_source_version=0.0.2, config.destination.source_specific_configs.victorops.application_field=set, catalog.destination_sync_mode.append=set, connector_destination_version=0.3.2, operation_count=0, config.source.page_size_for_large_streams=set, config.destination.source_specific_configs.statuspage.application_mapping=set, config.destination.edition_configs.hasura_admin_secret=set, connector_destination_definition_id=00000000-1111-2222-3333-444444444448}
2022-09-25 16:11:47 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-09-25 16:11:47 INFO i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2022-09-25 16:11:47 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-github-source:0.0.2 exists...
2022-09-25 16:11:48 INFO i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-github-source:0.0.2 was found locally.
2022-09-25 16:11:48 INFO i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:48 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-github-source-check-1-0-dkhgz --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-github-source:0.0.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-github-source:0.0.2 check --config source_config.json
2022-09-25 16:11:48 ERROR i.a.c.i.LineGobbler(voidCall):82 - WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:51 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):96 - Syncing `RepositoryStats` stream isn't available for repository `github/gitignore`.
2022-09-25 16:11:51 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):97 - Check succeeded
2022-09-25 16:11:51 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-09-25 16:11:51 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-09-25 16:11:51 INFO i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2022-09-25 16:11:51 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-faros-destination:0.3.2 exists...
2022-09-25 16:11:51 INFO i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-faros-destination:0.3.2 was found locally.
2022-09-25 16:11:51 INFO i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:51 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-faros-destination-check-1-0-ppsxq --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-faros-destination:0.3.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-faros-destination:0.3.2 check --config source_config.json
2022-09-25 16:11:51 ERROR i.a.c.i.LineGobbler(voidCall):82 - WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:55 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-09-25 16:11:55 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-09-25 16:11:55 INFO i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2022-09-25 16:11:55 INFO i.a.w.g.DefaultReplicationWorker(run):115 - start sync worker. job id: 1 attempt id: 0
2022-09-25 16:11:55 INFO i.a.w.g.DefaultReplicationWorker(run):127 - configured sync modes: {null.workflows=incremental - append, null.pull_requests=incremental - append, null.commits=incremental - append, null.issue_labels=full_refresh - append, null.assignees=full_refresh - append, null.organizations=full_refresh - append, null.issues=incremental - append, null.branches=full_refresh - append, null.issue_milestones=incremental - append, null.reviews=incremental - append, null.review_comments=incremental - append, null.pull_request_stats=incremental - append, null.releases=incremental - append, null.workflow_runs=full_refresh - append, null.users=full_refresh - append, null.collaborators=full_refresh - append, null.tags=full_refresh - append, null.repositories=full_refresh - append}
2022-09-25 16:11:55 INFO i.a.w.i.DefaultAirbyteDestination(start):69 - Running destination...
2022-09-25 16:11:55 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-faros-destination:0.3.2 exists...
2022-09-25 16:11:56 INFO i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-faros-destination:0.3.2 was found locally.
2022-09-25 16:11:56 INFO i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:56 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-faros-destination-write-1-0-qzsue --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-faros-destination:0.3.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-faros-destination:0.3.2 write --config destination_config.json --catalog destination_catalog.json
2022-09-25 16:11:56 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-github-source:0.0.2 exists...
2022-09-25 16:11:56 INFO i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-github-source:0.0.2 was found locally.
2022-09-25 16:11:56 INFO i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:56 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-github-source-read-1-0-nbdpg --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-github-source:0.0.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-github-source:0.0.2 read --config source_config.json --catalog source_catalog.json
2022-09-25 16:11:56 INFO i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):405 - Destination output thread started.
2022-09-25 16:11:56 INFO i.a.w.g.DefaultReplicationWorker(run):169 - Waiting for source and destination threads to complete.
2022-09-25 16:11:56 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):298 - Replication thread started.
2022-09-25 16:11:56 destination > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:56 source > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:59 source > Starting syncing SourceGithub
2022-09-25 16:11:59 source > Syncing `RepositoryStats` stream isn't available for repository `github/gitignore`.
2022-09-25 16:11:59 source > Syncing `Branches` stream isn't available for repository `github/gitignore`.
2022-09-25 16:11:59 source > 'github/gitignore'
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 129, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 120, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 101, in read
    stream_instances = {s.name: s for s in self.streams(config)}
  File "/airbyte/integration_code/source_github/source.py", line 186, in streams
    default_branches, branches_to_pull = self._get_branches_data(config.get("branch", ""), repository_args)
  File "/airbyte/integration_code/source_github/source.py", line 133, in _get_branches_data
    repo_branches = [default_branches[repo]]
KeyError: 'github/gitignore'
2022-09-25 16:11:59 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):335 - Total records read: 1 (0 bytes)

Deployment

• Local Deployment
• M1 MacBookPro 2022
• MacOS 12.6 - Monterey
• Docker Engine - 20.10.17
• Docker Desktop - 4.12.0

Error: Error response from daemon: create tmp\airbyte_local: "tmp\airbyte_local" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path

Describe the bug The GitHub canned source does not get PR diff stats.

The pull_request_stats stream (which exists in the community source AND for which we have a converter that updates the PR entityis deactivated because we get those errors:

2022-10-31 05:42:56 source > Syncing stream: pull_request_stats 
2022-10-31 05:43:13 source > Backing off _send(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://api.github.com/graphql, Response Code: 502, Response Text: {
   "data": null,
   "errors":[
      {
         "message":"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `F8DD:188C:158A262:164FCC8:635F6064` when reporting this issue."
      }
   ]
})
2022-10-31 05:43:13 source > Caught retryable error 'Request URL: https://api.github.com/graphql, Response Code: 502, Response Text: {
   "data": null,
   "errors":[
      {
         "message":"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `F8DD:188C:158A262:164FCC8:635F6064` when reporting this issue."
      }
   ]
}
' after 1 tries. Waiting 5 seconds then retrying...
2022-10-31 05:43:28 source > Backing off _send(...) for 10.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://api.github.com/graphql, Response Code: 502, Response Text: {
   "data": null,
   "errors":[
      {
         "message":"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `F8DD:188C:158B878:1651373:635F6076` when reporting this issue."
      }
   ]
})
2022-10-31 05:43:28 source > Caught retryable error 'Request URL: https://api.github.com/graphql, Response Code: 502, Response Text: {
   "data": null,
   "errors":[
      {
         "message":"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `F8DD:188C:158B878:1651373:635F6076` when reporting this issue."
      }
   ]
}
' after 2 tries. Waiting 10 seconds then retrying...

One shall investigate the nature of those errors in the source and figure out if this is a configuration issue, a transient issue, or something wrong with the source itself. Until then, the stream remains disabled and that data will be missing.

Describe the bug The use of field derived from a JSON-containing column makes import fail on field-level table sync check.

To Reproduce Steps to reproduce the behavior:

1. Start Faros
2. Import some tasks from Jira/GH/...
3. Trigger a database schema sync and field value scan
4. Check that tms_Task now has 2 versions of status category for tms_Task: the generated field statusCategory and the newly metabase-derived one status -> category
5. Create a dashboard with a non-SQL question using that status->category field as aggregation
6. Export the dashboard.
7. Try to re-import it.
8. Import fails on syncTable because the check can not find that field in the metadata.

Expected behavior Dashboard import from a template using such field is successful.

Screenshots and logs If applicable, add screenshots and logs to help explain your problem. Example of bad template that had to be fixed here

Notice the difference between the 2 field types:

Encountered during #231 .

Describe the bug The Faros Destination does not follow the sync mode spec (see https://github.com/faros-ai/airbyte-connectors/issues/575). Hence, all sync mode right now are either Full Refresh | Append or Incremental | Append which works as intended but not as an Airbyte user might expect. Once the Faros Destination is fixed, we will have to update the connections to use Overwrite.

Describe the bug refreshedAt is not updated when a record is modified.

To Reproduce Steps to reproduce the behavior:

1. Insert a record through GraphQL and check refreshedAt value
2. Update some record field and check refreshedAt value again

Expected behavior refreshedAt should be updated automatically on every record change.

Describe the bug In DORA dashboards, for the top left gauge chart Weekly Release / Deployment Frequency (Avg), we use a UI built query to compute the average. It is incorrect because weeks with no releases / deployments are simply ignored in the final averaging because of how the query is built. Indeed, the first aggregation that counts per week will have no entries for week with no release / deployment.

Crafting the proper query may (?) require changing the card to a SQL query.

Expected behavior If you have the following:

• week 1 - 1 release
• week 2 - 1 release
• week 4 - 1 release The Weekly Release frequency past 30 days should be 0.75, instead it is 1.

Screenshots and logs Notice how some weeks do not have releases.