Hi,

I'm running into an odd issue with the diff command, I'm hoping someone here can provide some guidance.

I'm following the guide Try the optic diff command, the command is successful if I compare two files in the same branch as demoed in Two files use case but when I run the diff command showned below across two different git branches I get an error stating that the object name 'C' is invalid.

The following command.

optic diff todo-api.yaml --base main

Results in the following error.

{
  stack: 'ResolverError: Command failed: git show C:Users\\circleupx\\source\\openapi-demo\\todo-api.yaml\n' +
    "fatal: invalid object name 'C'.\n" +
    ' Error opening file "Users\\circleupx\\source\\openapi-demo\\todo-api.yaml"\n' +
    '    at C:\\Users\\circleupx\\AppData\\Roaming\\npm\\node_modules\\@useoptic\\optic\\node_modules\\@useoptic\\openapi-io\\build\\parser\\resolvers\\git-branch-file-resolver.js:24:32\n' +
    '    at ChildProcess.exithandler (node:child_process:406:5)\n' +
    '    at ChildProcess.emit (node:events:520:28)\n' +
    '    at ChildProcess.emit (node:domain:475:12)\n' +
    '    at maybeClose (node:internal/child_process:1092:16)\n' +
    '    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)',
  code: 'ERESOLVER',
  message: 'Command failed: git show C:Users\\circleupx\\source\\openapi-demo\\todo-api.yaml\n' +
    "fatal: invalid object name 'C'.\n" +
    ' Error opening file "Users\\circleupx\\source\\openapi-demo\\todo-api.yaml"',
  source: 'Users\\circleupx\\source\\openapi-demo\\todo-api.yaml',
  path: null,
  toJSON: [Function: toJSON],
  name: 'ResolverError',
  footprint: 'null+Users\\circleupx\\source\\openapi-demo\\todo-api.yaml+ERESOLVER+Command failed: git show C:Users\\circleupx\\source\\openapi-demo\\todo-api.yaml\n' +
    "fatal: invalid object name 'C'.\n" +
    ' Error opening file "Users\\circleupx\\source\\openapi-demo\\todo-api.yaml"',
  toString: [Function: toString]
}

Doing some troubleshooting led me to the git-branch-file-resolver.js where I can see that the code is attempting to execute the command git show.

Based on my knowledge of the command git show, if you want to run git show on the same file across two different branches then you need to include the branch name in the git show command.

For example, currently, the optic CLI is attempting to execute the following command when I execute optic diff todo-api.yaml --base main

 git show C:Users\\circleupx\\source\\openapi-demo\\todo-api.yaml'

The command above should really be as follows.

 git show main:C\\Users\\circleupx\\source\\openapi-demo\\todo-api.yaml'

If update the git-branch-file-resolver.js file locally then I no longer the error "the object name 'C' is invalid".

I'm running Optic version 0.35.4.

Thank you.

Why

While most of the diffing already happens in Rust, the part that's involved with learning new bodies for undocumented endpoints is still in Scala, with the performance penalty that comes with it. Just was with the diffs, for reasonably sized projects this can take many minutes, while porting it to Rust took that back down to seconds.

What

Implements a new initial-bodies worker that calls to Rust instead of Scala.

A new learn command is added to the Rust CLI. This can be used from the Node.js side through learnUndocumentedBodies, which accepts the spec and interactions that are to be learned, returning a readable stream of endpoint bodies.

A new worker task uses this to implement the learning of an interaction per endpoint and method, just like the Scala version.

Like the Scala version, this implementation causes a re-diffing of the interaction. However, it's structured for that to change and go with a more efficient pipeline, where bodies are learned immediately after diff results come available from their first production. While changes on that scale are beyond the scope of this work, the path to it is open.

Processing the interactions (diffing + learning their shapes) happens in a similar concurrency model as the diffing, with multiple threads diffing interactions concurrently, sending results to a thread dedicated to results aggregation. If need be, results aggregation can be split up between multiple threads in the future, probably by partitioning by endpoint. For now, that's considered beyond the scope of this PR.

Validation

• [x] CI passes
• [x] Hidden behind a feature flag by default
• [x] No ScalaJS used on server + domain side.
• [x] Works as drop-in replacement for the UI client.

I'd like to serve the doc along with my API under the endpoint /doc but now for serving I need to run the api spec which is a node server that I have to host somehow.

Is it possible to generate html instead ?

Regards

Steps to reproduce:

1. run npm install optic-cli -g
2. run optic Got the error:

? Finish Install: Configure local Optic server. Yes
Installing local Optic server...
downloading and preparing JRE + Optic Server [====================================] 100% 0.0s
Optic server could not be installed.Smoketest failed.

Why

Slurp saves IHttpInteractions in batches to an S3 bucket. We want to load that data into your local Optic instance. This PR implements the S3 ingest capture saver that does that

So far so good!

Opens the UI and contains the right knowledge from the captures!

And we have a progress bar

https://user-images.githubusercontent.com/4368270/112524972-ca9ee380-8d76-11eb-99ec-81bec6e5d571.mov

TODO:

• [x] Better progress bar
• [x] Integrate with the existing CLI
• [x] Launch daemon and open browser upon completion
• [ ] Add option to limit total size of files ingested. E.g to 1gb or something.

So, i'm working on an API that will scale a lot over time, and when i was looking for a good OAs3 autodocumenting solution on SO, i stumbled with Optic.

I followed the install and setup guides and ended up with the next setup:

1. optic.yaml which initially had the suggested port: 4000, however i later changed it to port: 3300 since it was the one that Optic was trying to use, also take note on the command (more on that in 3.):

2. bin/www entry point, which differs from the examples in the guides, thus i tried different combinations with app and server objects:

3. And since this project is handled by PM2, note the npm run pm2 definition, it uses the PM2 Ecosystem Configuration File (see 4.):

4. Said file is just an autodocumented and easier to handle way to launch PM2 Processes, as far i understood Optic should end up intercepting/wrapping the app, which in time gets wrapped by PM2, however there might be a conflict around that matter:

These are the errors that i could not work around, no matter how i handle bin/www and optic.yaml:

I also tried just launching the api check start by using node app.js and node bin/www commands to completely avoid PM2, but the behavior remains.

I'm quite sure the problem is around 2....

I'm using node - express - typescript

After:

• installing optic
• typing api init in powershell in root directory of my API
• setting up all the options available on the page
• typing api start in powershell

I'm getting an error:

[optic] Optic needs more information about your API to continue.
This directory already has an optic.yml file.

Why

While most of the diffing already happens in Rust, the part that's involved with learning new affordances for shapes that caused diffs is still in Scala. These affordances power the UI where a user can choose to update their spec due to differences in fields. Since we're trying to get rid of Scala, it's important that we port this over

What

We're introducing a new worker that calls a new Learn command in the Rust diff engine CLI.

The learn CLI command now has a second subject implemented, shapes-diffs-affordances. This subject also requires --tagged-diff-results to point to the diff results that should be learned trails for. It then accepts interactions over stdin and learns from those one by one, for all results to be aggregated and returned over stdout.

The UI sends the diffId along with it's request for learning trail values, which allows the new Node.js worker to look up diff-results from disk. This replaces serialising and sending the diffs from the client to the server, which was an unnecessary overhead that's no longer required due to the shift of logic from UI -> backend.

A minimal amount of "re-diffing" is performed for each incoming interaction, basically to determine which interactions match a documented request or response body. From there, the qualifying interaction bodies are fully traversed and mapped, to then be matched against the trails relevant to the diffs. In the future, a more focused traverser could be implemented that only traverses the trail relevant to diffs, but so far, that kind of optimisation does not seem necessary.

Processing the interactions happens in a similar concurrency model as the diffing, with multiple threads diffing interactions concurrently, sending results to a thread dedicated to results aggregation. If need be, results aggregation can be split up between multiple threads in the future, probably by partitioning by endpoint. For now, that's considered beyond the scope of this PR.

Validation

• [x] CI passes
• [x] Hidden by a feature flag by default (OPTIC_RUST_SHAPE_DIFF_AFFORDANCES_LEARNER).
• [x] No ScalaJS involved on server + domain side.
• [x] Works as a drop-in replacement for the UI client. (small caveat: UI client adjusted to POST diffId to endpoint).

right now we are trying to intercept traffic of an environment that is rejecting because of self-signed certificate and disabling that check is not an option in that server, so please let me know if there is a way I can provide certs to be used instead of self-signed certificates being created in your code.

I am trying to generate a diff on a large capture and I tried a lot of times to run it but was unsuccessful in doing so because optic is getting killed.

I checked and monitored the ram, it is not a memory issue. I ran api debug:capture, it generated a 101 MB debug file.

I am uploading that here. Please let me know what I can do further to help you debug this issue.

I am trying to export as OpenAPI

Error: Cannot use GraphQLSchema "[object GraphQLSchema]" from another module or realm.

Ensure that there is only one instance of "graphql" in the node_modules
directory. If different versions of "graphql" are the dependencies of other
relied on modules, use "resolutions" to ensure only one version is installed.

https://yarnpkg.com/en/docs/selective-version-resolutions

Duplicate "graphql" modules cannot be used at the same time since different
versions may have different capabilities and behavior. The data from one
version used in the function from another could produce confusing and
spurious results.
    at instanceOf (/usr/local/lib/node_modules/@useoptic/cli/node_modules/@useoptic/spectacle/node_modules/graphql/jsutils/instanceOf.js:28:13)
    at isSchema (/usr/local/lib/node_modules/@useoptic/cli/node_modules/@useoptic/spectacle/node_modules/graphql/type/schema.js:36:34)
    at assertSchema (/usr/local/lib/node_modules/@useoptic/cli/node_modules/@useoptic/spectacle/node_modules/graphql/type/schema.js:40:8)
    at validateSchema (/usr/local/lib/node_modules/@useoptic/cli/node_modules/@useoptic/spectacle/node_modules/graphql/type/validate.js:44:28)
    at graphqlImpl (/usr/local/lib/node_modules/@useoptic/cli/node_modules/@useoptic/spectacle/node_modules/graphql/graphql.js:79:62)
    at /usr/local/lib/node_modules/@useoptic/cli/node_modules/@useoptic/spectacle/node_modules/graphql/graphql.js:28:31
    at new Promise (<anonymous>)
    at Object.graphql (/usr/local/lib/node_modules/@useoptic/cli/node_modules/@useoptic/spectacle/node_modules/graphql/graphql.js:26:10)
    at Object.queryWrapper (/usr/local/lib/node_modules/@useoptic/cli/node_modules/@useoptic/spectacle/build/spectacle.js:488:26)
    at Object.generateOpenApi (/usr/local/lib/node_modules/@useoptic/cli/node_modules/@useoptic/spectacle/build/openapi/index.js:7:38)

🍗 Description

Run your spectral configs within Optic. Some rulesets can run added, some always.

📚 References

Config:

spectral: 
   added:
     - ./latest-pagination.yml 
   always:
     - ./metadata.yml
     - ./operations.yml

👹 QA

How can other humans verify that this PR is correct?

🍗 Description

What does this PR do? Anything folks should know?

This PR implements optic api add. This can be called in two ways:

• optic api add - adds all detected openapi files in a repo
• optic api add <file_name> --history-depth n - adds a single api file with history up to a depth of n (0 is all history)

If token is in multiple orgs, we prompt for which org to add to. If a user does not select a ruleset, we prompt to see if they would like to create a standard or use an existing standard.

Also changed ruleset -> standard

TODO:

• [x] add notes for ruleset
• [x] write x-optic-url + x-optic-ruleset
• [x] add tests

TODO in follow up PR

• [ ] connect up APIs (fetch org from APIs, upload api, upload spec)

📚 References

Links to relevant docs (Notion, Twist, GH issues, etc.), if applicable.

👹 QA

How can other humans verify that this PR is correct?

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• [ ] fix(deps): update dependency ts-invariant to ^0.10.0
• [ ] chore(deps): update dependency micro to v10
• [ ] fix(deps): update dependency chalk to v5
• [ ] fix(deps): update dependency node-fetch to v3
• [ ] fix(deps): update dependency slice-ansi to v5 (slice-ansi, @types/slice-ansi)
• [ ] fix(deps): update dependency strip-ansi to v7
• [ ] fix(deps): update dependency url-join to v5

Detected dependencies

• [ ] Check this box to trigger a request for Renovate to run again on this repository

I have project with git submodule, that contains common definitions. When I run

optic diff src/openapi.yml --base c27e8b3

I get an error

{
  stack: 'ResolverError: Command failed: git show c27e8b3:src/shared/common.yml\n' +
    "fatal: path 'src/shared/common.yml' exists on disk, but not in 'c27e8b3'\n" +
    ' Error opening file "src/shared/common.yml"\n' +
    '    at /home/me/code/api/node_modules/@useoptic/openapi-io/build/parser/resolvers/git-branch-file-resolver.js:24:32\n' +
    '    at ChildProcess.exithandler (node:child_process:420:5)\n' +
    '    at ChildProcess.emit (node:events:513:28)\n' +
    '    at ChildProcess.emit (node:domain:489:12)\n' +
    '    at maybeClose (node:internal/child_process:1091:16)\n' +
    '    at Socket.<anonymous> (node:internal/child_process:449:11)\n' +
    '    at Socket.emit (node:events:513:28)\n' +
    '    at Socket.emit (node:domain:489:12)\n' +
    '    at Pipe.<anonymous> (node:net:313:12)',
  code: 'ERESOLVER',
  message: 'Command failed: git show c27e8b3:src/shared/common.yml\n' +
    "fatal: path 'src/shared/common.yml' exists on disk, but not in 'c27e8b3'\n" +
    ' Error opening file "src/shared/common.yml"',
  source: 'src/shared/common.yml',
  path: null,
  toJSON: [Function: toJSON],
  name: 'ResolverError',
  footprint: 'null+src/shared/common.yml+ERESOLVER+Command failed: git show c27e8b3:src/shared/common.yml\n' +
    "fatal: path 'src/shared/common.yml' exists on disk, but not in 'c27e8b3'\n" +
    ' Error opening file "src/shared/common.yml"',
  toString: [Function: toString]
}

where src/shared/ is mentioned submodule, and analysed openapi.yml references src/shared/common.yml.

Description - A few users have been asking about when OAS 2 support is coming. We were not sure how hard supporting multiple versions would be, but as we discovered when we released 3.1 support last month, the facts model effectively normalizes out the structure of the OpenAPI specifications.

The OAS 2 traverser will traverse the unique structure of these documents and map all of their types onto our standard facts. That mean OOTB our built-in rules will work. If a company has written custom rules that look at the raw values of different OpenAPI entities, those may break when encountering an unexpected spec version. That should be pretty rare. If you want to support both major versions, you need to update the custom rules anyway and that is not the worst way to find out you need to.

• replaces content contentType with produces and consumes
• replaces requestBody with the request parameter that has in: body
• removes nullable, oneOf, anyOf, allOf -- not supported in oas 2
• includes on some known limitations
  • omits formData and examples (these can come later if needed)
• adds OpenAPI v2 validation and semver parsing
• tests the facts produced by 3 OAS 2 specifications from Azure, & Petstore. Manually reviewed and snapshotted
  • looked for multiples of bodies for each produces and consumes
  • path parameters get generated properly
  • schema facts are correct

questions?

• All over our code we use ParseOpenAPIResult, and jsonLike is oas3 typed. That type here has never been correct. We say it is oas3, but there's no validation at this stage. The sourcemap parser will only fail if a $ref can't be resolved. It could be any data that is YAML and JSON. We have the validator, I propose that is where the typing gets applied. Ie validator would return `errors?: string[] , 3_0?: OpenAPI_V3, 3_1?: OpenAPI_V3_1, 2_0?: OpenAPI_V2_0
• Does the changelog or anything in Optic Cloud strongly expect the specs to be OpenAPI 3? What does the path to supporting OAS 2 look like in the changlog / cloud?
• What else could we write tests for?

current state?

• the validator is still set to fail on oas2, so we could merge these changes without the support being enabled
• traverser is working great. I can keep adding more tests.
• we can't turn it on until we address the ParseOpenAPIResult issue and do some retyping.
• we can't turn it on until we validate Changelog works OR decide not to support that to start
  • what if we choose not to upload oas2 to cloud / changelogs? That lets us give this to customers and validate interest in full story before going further, and takes pressure off the cloud team who have a lot of other things going on right now

Security schemas are used to define the authentication for the operations in OpenAPI. They are shorthand for headers like Authentication or query parameters that include a token.

https://spec.openapis.org/oas/v3.1.0#security-scheme-object

There's a discussion going on here to define which security schema changes we consider breaking https://github.com/opticdev/optic/discussions/1305