// unified-latex-util-parse

• npm install
• npm run build

../unified-latex-util-match/dist/index.d.ts:3:1127 - error TS2307: Cannot find module '../unified-latex-types/dist' or its corresponding type declarations.

3 export declare const anyEnvironment: (node: any) => node is import("../unified-latex-types/dist").Environment, anyMacro: (node: any) => node is import("../unified-latex-types/dist").Macro, anyString: (node: any) => node is import("../unified-latex-types/dist").String, argument: (node: any) => node is import("../unified-latex-types/dist").Argument, blankArgument: (node: any) => boolean, comment: (node: any) => node is import("../unified-latex-types/dist").Comment, environment: (node: any, envName?: string | undefined) => node is import("../unified-latex-types/dist").Environment, group: (node: any) => node is import("../unified-latex-types/dist").Group, macro: (node: any, macroName?: string | undefined) => node is import("../unified-latex-types/dist").Macro, math: (node: any) => node is import("../unified-latex-types/dist").InlineMath | import("../unified-latex-types/dist").DisplayMath, parbreak: (node: any) => node is import("../unified-latex-types/dist").Parbreak, string: (node: any, value?: string | undefined) => node is import("../unified-latex-types/dist").String, whitespace: (node: any) => node is import("../unified-latex-types/dist").Whitespace; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The distribution packages on npmjs are missing the index.cjs files. The files are explicitly exported in all the package.json file, e.g.,

https://github.com/siefkenj/unified-latex/blob/main/packages/unified-latex-util-parse/package.json#L28

I think the only reason the cjs files aren't in the packages is because

https://github.com/siefkenj/unified-latex/blob/main/scripts/make-package.mjs#L49

doesn't include "**/*.cjs".

That's a guess since I wasn't able to build from source yet; otherwise, I would make a PR.

What exact version of nodejs and npm are you using for development? I'm using

~/unified-latex$ node --version
v14.19.2
~/unified-latex$ npm --version
8.10.0

on Ubuntu 20.04.

Hi I'm trying to get webpack to work with unified-latex.

Here is my webpack.config.js:

const nodeExternals = require("webpack-node-externals");
const path = require("path");

module.exports = {
  name: 'server',
  entry: {
    server: path.resolve(__dirname, 'server/server.tsx'),
  },
  mode: 'production',
  output: {
    path: path.resolve(__dirname, 'dist'),
    filename: '[name].js',
  },
  resolve: {
    extensions: ['.ts', '.tsx'],
  },
  externals: [nodeExternals()],
  target: 'node',
  node: {
    __dirname: false,
  },
  module: {
    rules: [
      {
        test: /\.tsx?$/,
        loader: 'ts-loader',
        options: {
          compilerOptions: {
            "noEmit": false
          }
        },
      },
      {
        test: /\.(sa|sc)ss$/,
        use: [
          {
            loader: "css-loader",
            options: {
              importLoaders: 2
            }
          },
          {
            loader: "sass-loader"
          }
        ]
      },
      {
        test: /\.svg$/,
        loader: 'url-loader',
        options: {
          limit: 8192,
        }
      },
    ],
  },
}

Here is my tsconfig.json:

{
  "compilerOptions": {
    "target": "es5",
    "lib": [
      "dom",
      "dom.iterable",
      "esnext"
    ],
    "allowJs": false,
    "skipLibCheck": true,
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "strict": true,
    "forceConsistentCasingInFileNames": true,
    "noFallthroughCasesInSwitch": true,
    "module": "esnext",
    "moduleResolution": "node",
    "resolveJsonModule": true,
    "isolatedModules": true,
    "noEmit": true,
    "jsx": "react-jsx"
  },
  "include": [
    "src"
  ]
}

After that I try to run:

node dist/server.js

And I got the following error:

internal/modules/cjs/loader.js:1102
      throw new ERR_REQUIRE_ESM(filename, parentPath, packageJsonPath);
      ^

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /app/node_modules/unified/index.js
require() of ES modules is not supported.
require() of /app/node_modules/unified/index.js from /app/node_modules/@unified-latex/unified-latex-util-parse/index.cjs is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /app/node_modules/unified/package.json.

    at new NodeError (internal/errors.js:322:7)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1102:13)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (/app/node_modules/@unified-latex/unified-latex-util-parse/index.cjs:39:22)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32) {
  code: 'ERR_REQUIRE_ESM'
}

Seems like package unified-latex-util-parse mix cjs and js in imports. I have tried different approaches, but have achieved a positive result.

Also it seems like "@unified-latex/unified-latex-to-hast": "^1.2.0" has missing .d.ts in npm, but this problem was fixed using following stupid fix:

types-fix.d.ts:

declare module "@unified-latex/unified-latex-to-hast" {
    export function convertToHtml(tree: Ast.Node | Ast.Node[]): string;
};

This can be fixed switching to version 1.1.0 of @unified-latex/*.

Hi there, super excited about this!

I wanted to point out a potential bug. In beamer documents, it's common to use \begin{frame}{title} or \begin{frame}{title}{subtitle}. This doesn't work in the latex-parser-playground.

This code:

\documentclass{beamer}

\begin{document}

\begin{frame}{Title}

  \begin{itemize}
    Test
  \end{itemize}

\end{frame}

\end{document}

results in this:

\documentclass{beamer}

\begin{document}
	\begin{frame}
		{Title}

		\begin{itemize}
			Test
		\end{itemize}
	\end{frame}
\end{document}

Notice how the {Title} is on a separate line

Its index.d.ts contains a line:

export declare const anyEnvironment: (node: any) => node is import("@unified-latex/unified-latex-types/dist").Environment, anyMacro: (node: any) => node is import("@unified-latex/unified-latex-types/dist").Macro, anyString: (node: any) => node is import("@unified-latex/unified-latex-types/dist").String, argument: (node: any) => node is import("@unified-latex/unified-latex-types/dist").Argument, blankArgument: (node: any) => boolean, comment: (node: any) => node is import("@unified-latex/unified-latex-types/dist").Comment, environment: (node: any, envName?: string | undefined) => node is import("@unified-latex/unified-latex-types/dist").Environment, group: (node: any) => node is import("@unified-latex/unified-latex-types/dist").Group, macro: (node: any, macroName?: string | undefined) => node is import("@unified-latex/unified-latex-types/dist").Macro, math: (node: any) => node is import("@unified-latex/unified-latex-types/dist").InlineMath | import("@unified-latex/unified-latex-types/dist").DisplayMath, parbreak: (node: any) => node is import("@unified-latex/unified-latex-types/dist").Parbreak, string: (node: any, value?: string | undefined) => node is import("@unified-latex/unified-latex-types/dist").String, whitespace: (node: any) => node is import("@unified-latex/unified-latex-types/dist").Whitespace;

However @unified-latex/unified-latex-types as published in npm doesn't contain a directory dist.

I using this library in following way:

import { parse } from "@unified-latex/unified-latex-util-parse";
import { convertToHtml } from "@unified-latex/unified-latex-to-hast";

const ast = parse(content ?? "");
const html = convertToHtml(ast);

I want to add support for custom implementation for macro:

<<Some text>> % this should be generated like &laquo;Some text&raquo;
% It looks like this is a common feature and should be added to this library.

\includegraphics{url} % this should be generated like <img src="PREFIX/url" />
% It looks like it's a custom implementation, at least for html generation.

How can I add this support? It seems I should add a macro to parse and an html generator to convertToHtml, but the current interface doesn't allow this.

Sorry for clogging up your issues, but I just quickly wanted to thank you for this creating and maintaining this project! I tried to do something similar a couple of months ago here and here but your implemenation seems much more robust! I'll be switching to your version at some point, so thank you!

I'm trying to write some custom plugins for transforming latex, and so far, I really liked your work, especially in how you provide every required toolsets via modular package structure. While experimenting with it, I noticed that currently there's no support for macros with default arguments.

% input.tex
\documentclass{article}
% \newcommand\foo[2][bar]{#1,#2}
\begin{document}
    \foo[bar]{baz}
    \foo{baz}
\end{document}

unified-latex input.tex -e "\\newcommand\\foo[2][bar]{#1,#2}"

Then the result will be

\documentclass{article}

%\newcommand\foo[2][bar]{#1,#2}

\begin{document}
        bar,baz

        ,baz
\end{document}

where the expected output would have two identical lines of bar,baz.

In unified-latex-util-macros, an xpase-style signature is computed, but it produces o irrespective of the presence of a default argument. I've also noticed that manually passing O{bar} m via \NewDocumentCommand still produces unexpected results, so there should be another issue regarding macro expanders.

It'd be nice if this support can be added. I'm wondering to know if there's any blocker for implementing it, I'm interested in sending a PR.

Hi @siefkenj -- just wanted to say thanks for this library, I just started using it to parse some latex documents, working great right now, I am sure I will have some questions as I get into it more.

I also wanted to introduce myself 👋 and a project I am working on, where there might be a chance to collaborate / coordinate a bit with this project if you all are game!?

I am working for the @executablebooks project on MyST Markdown, which is a markup language that is gaining traction in the python communities through tools like JupyterBook, Sphinx (as it is based on RST). We are currently working on standardizing some of the AST underlying MyST, initial work is here: https://spec.myst.tools/

This allows some pretty cool web-based rendering of things like cross-references and all sorts of other citations (e.g. see typography for inline demos). We have also started working on creating various latex templates so that you can take your markup and write it to latex using one of a few hundred journals. We have also just started pushing into JATS export (on every inline demo in the docs), which is used in scientific publishing and archiving.

All of this is one way at the moment (with the upcoming exception of JATS):

What would be amazing would be to have some interop with unified-latex to support reading (and probably in the future better/prettier writing) of latex documents.

I am not quite sure what next steps would be for that, I would be happy to meet and share our project's vision/goals? I am mostly here to show enthusiasm for your project. 🚀 :)

Parsing

$x

y$

produces x followed by two whitespaces followed by y. All whitespace should be collapsed (otherwise it can produce a blank line when pretty-printing).

Currently, aside from recognizing groups and environments, math is treated as a simple stream of tokens. As LaTeX is known for its excellent math equation generation, better parsing of math allows better formatting of math expressions, easing maintenance of documentation websites making heavy use of math notation.

Here is a rough idea on how to implement it, inspired in part by programming language parsers:

expr_unit "expression unit"
    = num+
    / char
    / whitespace
    / !binary_macro macro
    / begin_group b_expr end_group

power "superscript and subscript"
    = expr_unit
    / power "^" expr_unit
    / power "_" expr_unit

d_expr "delimited expression"
    = power
    / left_delimiter b_expr? right_delimiter

m_expr "multiplication expression"
    = power
    / m_expr power

u_expr "unary plus/minus expression"
    = m_expr
    / [+-] u_expr

b_expr "binary expression"
    = u_expr
    / b_expr binary_operator u_expr

A description in table form for the commons:

| Operator | Description | | :------------------------ | :----------------------------------- | | Superscript and subscript | Normally ^ and _. | Logical delimitation | Parentheses, brackets, \{, \}, and \left-\right pairs. The delimiter on the left and right can be different. | "Multiplication" | Denoted by the lack of operators between two logical units. 2x is considered a "multiplication" expression, while 2 \times x is not. | Unary operators | +, -, \neg, etc. | Binary operators | +, -, =, \times, \le, etc.