Grammar file: https://github.com/opencybersecurityalliance/kestrel-lang/blob/develop/src/kestrel/syntax/kestrel.lark

When I tried to parse proc2 = GET incomplete statement, the parser does not provide the expected token type for next token when I caught the exception UnexpectedToken.

Expected result: Based on the grammar

statement: VARIABLE "=" command
         | command
         
// "?" at the beginning will inline command
?command: get
        | find
        | disp
        | info
        | apply
        | join
        | sort
        | group
        | load
        | save
        | new
        | merge
get: "get"i ENTITY_TYPE ("from"i DATASRC)? "where"i STIXPATTERNBODY (starttime endtime)?

it should provide the next expected token type ENTITY_TYPE, so we can do something afterwards.

Grammar file: https://github.com/opencybersecurityalliance/kestrel-lang/blob/release/src/kestrel/syntax/kestrel.lark Generated parser: kestrelParser.js.zip

When parsing this statement var=get, the parser throws the unexpected token exception with

e.line =1
e.column=5

However, the column should be 7.

Same incorrect column info for the following test strings. var=get file, e.column is 7, but should be 12. var=get file from, e.column is 14, but should be 17. var=get file from abc, e.column is 19, but should be 21.

When I tried lark-js to generate the parser from .lark file, the parser includes "_plugins": {} in to the parser file which resulted to the console error.

Grammar file: https://github.com/opencybersecurityalliance/kestrel-lang/blob/develop/src/kestrel/syntax/kestrel.lark

Grammar file: https://github.com/opencybersecurityalliance/kestrel-lang/blob/develop/src/kestrel/syntax/kestrel.lark Generated parser: kestrelParser.js.zip

Code

function handle_errors(e) { return true; }

  try {
    treeData = parser.parse("proc2 =", null, handle_errors).children[0];
  } catch (e) {
    console.debug("uncaught error:", e)
  }

Expected: can stop parsing and get error info.

Closes #8.

I know you're running two equivalent packages, lark and lark-parser, but the suggested name over in lark-parser/lark is just lark, so I changed that here too.

Grammar file: https://github.com/opencybersecurityalliance/kestrel-lang/blob/develop/src/kestrel/syntax/kestrel.lark Generated parser: kestrelParser.js.zip

Code to parse string disp a\nproc2 = get. The 1st line disp a is valid string. The 2nd line proc2=get is invalid and should return error. However, the exception I got from parser told me the error occurs at line 1. I checked the parser, and it uses \n as newline char, so it should return line 2 has error.

function handle_errors(e) { return true; }

  try {
    treeData = parser.parse("disp a\nproc2 = get", null, handle_errors).children[0];
  } catch (e) {
    console.debug("uncaught error:", e)
  }

Expected: the exception should return correct line number where the error occurs.

Grammar file: https://github.com/opencybersecurityalliance/kestrel-lang/blob/develop/src/kestrel/syntax/kestrel.lark Based on the grammar

statement: VARIABLE "=" command
         | command
         
// "?" at the beginning will inline command
?command: get
        | find
        | disp
        | info
        | apply
        | join
        | sort
        | group
        | load
        | save
        | new
        | merge
get: "get"i ENTITY_TYPE ("from"i DATASRC)? "where"i STIXPATTERNBODY (starttime endtime)?

The from and where are keywords for get command. However, when I parsed this statement "procs2 = GET process FROM procs WHERE [process:pid = 10578]" The parsed tree is as below which does not contain the from and where keywords. Is there any options I can pass to the parser to get those keywords in the parse tree?

{
  "data": "procs2 = GET process FROM procs WHERE [process:pid = 10578]",
  "children": [
    {
      "type": "VARIABLE",
      "start_pos": 0,
      "value": "procs2",
      "line": 1,
      "column": 1,
      "end_line": 1,
      "end_column": 7,
      "end_pos": 6
    },
    {
      "data": "get",
      "children": [
        {
          "type": "ENTITY_TYPE",
          "start_pos": 13,
          "value": "process",
          "line": 1,
          "column": 14,
          "end_line": 1,
          "end_column": 21,
          "end_pos": 20
        },
        {
          "type": "DATASRC",
          "start_pos": 26,
          "value": "procs",
          "line": 1,
          "column": 27,
          "end_line": 1,
          "end_column": 32,
          "end_pos": 31
        },
        {
          "type": "STIXPATTERNBODY",
          "start_pos": 38,
          "value": "[process:pid = 10578]",
          "line": 1,
          "column": 39,
          "end_line": 1,
          "end_column": 60,
          "end_pos": 59
        }
      ],
      "_meta": {
        "empty": false,
        "line": 1,
        "column": 10,
        "start_pos": 9,
        "container_line": 1,
        "container_column": 10,
        "end_line": 1,
        "end_column": 60,
        "end_pos": 59,
        "container_end_line": 1,
        "container_end_column": 60
      }
    }
  ],
  "_meta": {
    "empty": false,
    "line": 1,
    "column": 1,
    "start_pos": 0,
    "container_line": 1,
    "container_column": 1,
    "end_line": 1,
    "end_column": 60,
    "end_pos": 59,
    "container_end_line": 1,
    "container_end_column": 60
  },
  "type": "statement"
}

lark-js 0.0.7 is incompatible with the newly-released lark 0.12.0.

This is because poetry follows semver clause 4 when you use the ^ operator to define dependencies, and treats v0.Y.Z as unstable.

As an example, while people correctly expect that ^1.2.3 is also be compatible with 1.3.0, this doesn't apply to packages with a major version of 0. In the case of a version like ^0.Y.Z, the only compatible versions are those which only change Z.

I'm not sure if you intended on also updating and releasing pypi packages of lark-js when new versions of lark are released, à la the live code thing, but as it stands - I can't use lark 0.12 with lark-js. I see that you're working on a v1 release for lark, which is great, but until then, maybe we can use the inequality >= dependency specifiers?

Generated parser: kestrelParser.zip

Code

function handle_errors(e) { return true; }

  try {
    treeData = parser.parse("proc2 =", null, handle_errors).children[0];
  } catch (e) {
    console.debug("uncaught error:", e)
  }

Expected: can get treeData after parsing even if the statement is incomplete.

Fix the compiler errors addressed in https://github.com/lark-parser/Lark.js/issues/28. Other changes are to remove the extra spaces which are made automatically by IDE.

• Added support for bundling to: Browser, Node commonJS, Node ESM
• Volta support
• Added files field to minimize bundled files
• Fix to isMap, is_array, and differ Node and browser version by bundling instead of checking (Needed to ESM support)

TODO

• [ ] Required final NPM package name
• [ ] Adding details to package.json such as author (you) and repo (this)
• [ ] Testing bundles on Node 12 + other bundlers for browser version

Generated parser file (please unzip it first): kestrelParser.js.zip

We were using React 17.0.2, Next 12.1.0 and the parser can be imported and created successfully.

const parser = get_parser({ keep_all_tokens: true });

But after upgrading to React 18.2.0, Next 12.3.2, there is an error while running in the generated parser like below.

Do you have any ideas how to solve the issue? Thanks!

Hi, Let's say I use https://github.com/crytic/amarna/blob/main/amarna/grammars/cairo.lark and https://github.com/lark-parser/Lark.js to get the AST tree of a cairo file. What do I use after to generate a cairo file from the AST ? Thank you

I've noticed some errors when using a terminal "production" rule of the form

T0: T1 | T2 | T3

where all of the given expressions are terminals. These errors only occur in the standalone parser generated by Lark.js; the same grammar will correctly parse an identical string in the python version of lark. I've isolated two hopefully-minimal-enough example cases below.

This seems to be similar to #21 in that it's related to some Javascript-specific regex foible that gets encountered when agglomerating terminals together via |, but as I'm not super-familiar with the internals of the library I can't be sure. As in #21, replacing VALUE with value everywhere (i.e. replacing the terminal rule with a non-terminal one) causes both of the following examples to parse correctly.

Example 1

This grammar:

?start: thing
thing: thing W thing
    | expr
expr: label W? VALUE
    | VALUE
label: BARE_WORD W? ":"
W: /[ \t\n\v\f]/+
VALUE: NUMBER | BARE_WORD | STRING
BARE_WORD: /[^\s:\(\)]/+
STRING: "\"" /((?:\\"|[^\r\n"]))/* "\""
NUMBER: /[0-9]+/

fails with UnexpectedToken when attempting to parse the string "a:b", although running it in the Python version of Lark results in a correct parse.

Example 2

This grammar:

?start: thing
thing: label VALUE | VALUE
label: BARE_WORD W? ":"
W: /[ \t\n\v\f]/+
VALUE: NUMBER | BARE_WORD | STRING
BARE_WORD: /[^\s:\(\)]/+
STRING: "\"" /((?:\\"|[^\r\n"]))/* "\""
NUMBER: /[0-9]+/

fails with SyntaxError: Invalid flags supplied to RegExp constructor 'nully' during lexing of the same string "a:b"; the Python version also correctly parses it.

Trying to output a js parser for this matter.lark file fails with the following error:

$ ~/.local/bin/lark-js matter_grammar.lark -o matter_grammar.js
...
lark.exceptions.GrammarError: Using an undefined rule: NonTerminal('start')

Yet solving this is as simple as passing the correct start symbol with the -s flag with idl start symbol used by this .lark file:

$ ~/.local/bin/lark-js matter_grammar.lark -s idl -o matter_grammar.js

The lark-js should advise the user that the .lark file has no start symbol, and that the correct start symbol should be passed via --start rather than dumping a raw error backtrace.

Grammar file: https://github.com/opencybersecurityalliance/kestrel-lang/blob/develop/src/kestrel/syntax/kestrel.lark Generated parser: kestrelParser.js.zip

When parsing this statement procs2 = GET process abc, the parser throws the exception like below which is not caught by parser.

Code

function handle_errors(e) { return true; }

  try {
    treeData = parser.parse(text, null, handle_errors).children[0];
  } catch (e) {
    console.debug("uncaught error:", e)
  }

Expected: This kind of error can be handled by the parser, so we can get the parsing tree and the error info like Unexpected character or Unexpected Token.

I'm including a generated parser.js as part of a large web project. Webpack with the Terser plugin is configured to minify all of the javascript. The default (I believe) behavior is to rename all non-top-level classes to one or two-letter names.

But serialized data, like this:

  0: {
    name: 'NUMBER',
    pattern: {
      value:
        '(?:(?:(?:[0-9])+(?:e|E)(?:(?:\\+|\\-))?(?:[0-9])+|(?:(?:[0-9])+\\.(?:(?:[0-9])+)?|\\.(?:[0-9])+)(?:(?:e|E)(?:(?:\\+|\\-))?(?:[0-9])+)?)|(?:[0-9])+)',
      flags: [],
      _width: [1, 4294967295],
      __type__: 'PatternRE',
    },
    priority: 0,
    __type__: 'TerminalDef',
  },

includes names of classes as string literals.

When the classes are renamed, the data can no longer be deserialized and an exception is thrown.

Can a dictionary be built that maps the serialized type names to classes without relying on the class names themselves?