Unified-myst is a monorepo containing packages for using MyST

Overview

unified-myst (IN-DEVELOPMENT)

unified-myst is a monorepo containing packages for using MyST (Markedly Structured Text), within the unified ecosystem. It will eventually inform myst-spec.

It currently contains packages for:

All packages are scoped under the @unified-myst namespace: https://www.npmjs.com/org/unified-myst

Example

Take the target syntax (name)=.

The @unified-myst/target-extension package provides the mystTargetMmarkExt, mystTargetHtmlExt, and mystTargetMdastExt extensions.

First we can parse the target syntax to micromark-tokens:

import { parse } from 'micromark/lib/parse'
import { postprocess } from 'micromark/lib/postprocess'
import { preprocess } from 'micromark/lib/preprocess'
import { mystTargetMmarkExt } from '@unified-myst/target-extension'

const options = { extensions: [mystTargetMmarkExt] }
const events = postprocess(
    parse(options).document().write(preprocess()('(name)=', 'utf8', true))
)
console.log(
    events.map((event) => {
        return [
            event[0],
            event[1].type,
            `${event[1].start.line}.${event[1].start.column}`,
            `${event[1].end.line}.${event[1].end.column}`,
        ]
    })
)
// [
//   [ 'enter', 'mystTarget', '1.1', '1.8' ],
//   [ 'enter', 'mystTargetMarker', '1.1', '1.2' ],
//   [ 'exit', 'mystTargetMarker', '1.1', '1.2' ],
//   [ 'enter', 'mystTargetString', '1.2', '1.6' ],
//   [ 'enter', 'data', '1.2', '1.6' ],
//   [ 'exit', 'data', '1.2', '1.6' ],
//   [ 'exit', 'mystTargetString', '1.2', '1.6' ],
//   [ 'enter', 'mystTargetMarker', '1.6', '1.8' ],
//   [ 'exit', 'mystTargetMarker', '1.6', '1.8' ],
//   [ 'exit', 'mystTarget', '1.1', '1.8' ]
// ]

Tokens are pointers to slices of the original source text, which can be nested:

micromark tokens

These tokens can be compiled directly to HTML:

🔗">
import { micromark } from 'micromark'
import { mystTargetMmarkExt, mystTargetHtmlExt } from '@unified-myst/target-extension'

const options = {
    extensions: [mystTargetMmarkExt],
    htmlExtensions: [mystTargetHtmlExt],
}
console.log(micromark('(name)=', options))
// 🔗

Or be compiled to MDAST-nodes:

import { fromMarkdown } from 'mdast-util-from-markdown'
import { mystTargetMmarkExt, mystTargetHtmlExt } from '@unified-myst/target-extension'

const options = {
    extensions: [mystTargetMmarkExt],
    mdastExtensions: [mystTargetMdastExt],
}
const mdast = fromMarkdown('(name)=', 'utf8', options)
console.log(JSON.stringify(mdast, null, '  '))
// {
//   "type": "root",
//   "children": [
//     {
//       "type": "target",
//       "position": {
//         "start": {
//           "line": 1,
//           "column": 1,
//           "offset": 0
//         },
//         "end": {
//           "line": 1,
//           "column": 8,
//           "offset": 7
//         }
//       },
//       "label": "name"
//     }
//   ],
//   "position": {
//     "start": {
//       "line": 1,
//       "column": 1,
//       "offset": 0
//     },
//     "end": {
//       "line": 1,
//       "column": 8,
//       "offset": 7
//     }
//   }
// }

Development

The repository is a monorepo, with shared resources and dependencies. To manage the packages and run script commands against one or more packages, it is advised to use lerna, or using the npm workspace commands.

The repository is currently typed using JSDoc, as opposed to TypeScript. This decision was made, primarily because core packages in the unified ecosystem are also typed in this way, and also just because I wanted to try it out. See this post for a brief comparison. The repository may be converted to TypeScript in the future, but for now it is quite helpful for development, to not require a compilation step.

The repository is formatted with prettier, linted with eslint, and unit tested using Jest. There are a number of JavaScript unit testing frameworks (see this comparison, but jest was chosen because of it is easy to setup/use, flexible, and well used in large projects. See this guide for steps required for ECMAScript Modules (ESM) support. Snapshot Testing is also used extensively; to update snapshots, run lerna run test -- -- -u or npm run -ws test -- -u.

rollup.js is used for package bundling. Note that most packages in the unified ecosystem are ESM only

To release new versions of the packages, run lerna publish.

Comments
  • How to properly build/publish packages with internal dependencies

    How to properly build/publish packages with internal dependencies

    In @unified-myst/core-parse it has internally mapped dependencies: https://github.com/executablebooks/unified-myst/blob/d77a67e1a87104910a51fe0599cc738bc419f356/packages/core-parse/package.json#L56-L62

    It then has a rollup config for building: https://github.com/executablebooks/unified-myst/blob/d77a67e1a87104910a51fe0599cc738bc419f356/packages/core-parse/rollup.config.js#L1

    Firstly, when this gets published (va lerna) as an ESM module, these internal mappings are not "transformed" in any way. This means if one simply runs npm install @unified-myst/core-parse, it will not install any of these dependencies.

    Secondly, trying to build a "standalone" bundle, I haven't yet worked out how to get @rollup/plugin-node-resolve to resolve these internal mappings.

    Finally both lerna run build/lerna publish and npm run build -ws seem to not do any resolution of the order of building/publishing, e.g. that core-parses internal dependencies must be built before building core-parse itself

    opened by chrisjsewell 5
  • ✨ NEW: Add `targetExtension`

    ✨ NEW: Add `targetExtension`

    This extension adds the propagateTargets transform, to propagate target labels to the next viable node. For example:

    import { Processor } from '@unified-myst/core-parse'
    import * as ext from '@unified-myst/core-parse/extensions'
    import { removePosition } from 'unist-util-remove-position'
    import { dump } from 'js-yaml'
    
    const parser = new Processor()
        .use(ext.syntaxMystExtension)
        .use(ext.figureExtension)
        .use(ext.targetExtension)
    
    let text = `
    (target1)=
    % comment
    (target2)=
    # header
    
    - (target3)=
      paragraph
    
    (target4)=
    \`\`\`{image} a.png
    :name: other
    \`\`\`
    
    (target5)=
    `
    
    let ast = parser.toAst(text).ast
    ast = removePosition(ast, true)
    
    console.log(dump(ast))
    

    Results in:

    type: root
    children:
      - type: mystTarget
        label: target1
      - type: mystComment
        value: ' comment'
      - type: mystTarget
        label: target2
      - type: heading
        depth: 1
        children:
          - type: text
            value: header
        mystIds:
          - target1
          - target2
      - type: list
        ordered: false
        start: null
        spread: false
        children:
          - type: listItem
            spread: false
            checked: null
            children:
              - type: mystTarget
                label: target3
              - type: paragraph
                children:
                  - type: text
                    value: paragraph
                mystIds:
                  - target3
      - type: mystTarget
        label: target4
      - type: mystDirective
        name: image
        args:
          - a.png
        options:
          name: other
        value: ''
        bodyOffset: 2
        children:
          - type: image
            url: a.png
            label: other
            mystIds:
              - other
              - target4
      - type: mystTarget
        label: target5
    

    and produces the warning:

    {
      message: 'No node found to propagate target label to: target5',
      level: 'warning',
      position: {
        start: { line: 14, column: 1, offset: 102 },
        end: { line: 14, column: 11, offset: 112 }
      },
      type: 'propagateTargets'
    }
    
    opened by chrisjsewell 2
  • ✨ NEW: Add process logger

    ✨ NEW: Add process logger

    The logger is passed to all role/directive processors and event hooks.

    Any logger object can be initialized on the processor, which defines the logger interface, and the logger can have attached observers.

    opened by chrisjsewell 2
  • ✨ NEW: Add `linkReferenceExtension`

    ✨ NEW: Add `linkReferenceExtension`

    Extract all definitions and replace linkReference with their definitions. Example:

    import { Processor } from '@unified-myst/core-parse'
    import * as ext from '@unified-myst/core-parse/extensions'
    import { removePosition } from 'unist-util-remove-position'
    import { dump } from 'js-yaml'
    
    const parser = new Processor().use(ext.linkReferenceExtension)
    let text = `
    [*Hi*][a]
    
    [a]: https://example1.com
    [a]: https://example2.com
    `
    let ast = parser.toAst(text).ast
    ast = removePosition(ast, true)
    
    console.log(dump(ast))
    

    Generates:

    type: root
    children:
      - type: paragraph
        children:
          - type: link
            url: https://example1.com
            title: null
            children:
              - type: emphasis
                children:
                  - type: text
                    value: Hi
    

    and logs warning:

    {
      message: 'Duplicate definition identifier: a',
      level: 'warning',
      position: {
        start: { line: 6, column: 1, offset: 39 },
        end: { line: 6, column: 26, offset: 64 }
      },
      type: 'definition'
    }
    
    opened by chrisjsewell 0
  • ✨ NEW: Add `tableExtension`

    ✨ NEW: Add `tableExtension`

    import { Processor } from '@unified-myst/core-parse'
    import * as ext from '@unified-myst/core-parse/extensions'
    import { removePosition } from 'unist-util-remove-position'
    import { dump } from 'js-yaml'
    
    const parser = new Processor()
        .use(ext.syntaxMystExtension)
        .use(ext.tableExtension)
    
    let text = `
    \`\`\`{table} This is a caption
    :name: test
    :class: a
    
    | a | b |
    | - | - |
    | c | d |
    \`\`\`
    `
    
    let ast = parser.toAst(text).ast
    ast = removePosition(ast, true)
    
    console.log(dump(ast))
    

    Creates:

    type: root
    children:
      - type: mystDirective
        name: table
        args:
          - This is a caption
        options:
          name: test
          class:
            - a
        value: |-
          | a | b |
          | - | - |
          | c | d |
        bodyOffset: 4
        children:
          - type: container
            kind: table
            children:
              - type: caption
                children:
                  - type: text
                    value: This is a caption
              - type: table
                align:
                  - null
                  - null
                children:
                  - type: tableRow
                    children:
                      - type: tableCell
                        children:
                          - type: text
                            value: a
                      - type: tableCell
                        children:
                          - type: text
                            value: b
                  - type: tableRow
                    children:
                      - type: tableCell
                        children:
                          - type: text
                            value: c
                      - type: tableCell
                        children:
                          - type: text
                            value: d
            classes:
              - a
            label: test
            mystIds:
              - test
    
    opened by chrisjsewell 0
  • Abbreviation: (`explanation` or `title`)

    Abbreviation: (`explanation` or `title`)

    Realize that the sphinx world abbreviation has "explanation", in mdast as well as HTML title is more common? https://github.com/executablebooks/unified-myst/blob/main/packages/core-parse/src/extensions/inlineMarkup.js#L47

    Myst node is currently defined here: https://executablebooks.github.io/myst-spec/myst.schema.html#abbreviation

    opened by rowanc1 0
  • Suggestion for an core-parse/extensions PR?

    Suggestion for an core-parse/extensions PR?

    I am looking to contribute an extension here to get a feel for how it works, which should I choose?

    There is a decent list of features here: https://github.com/executablebooks/mystjs#mystjs-features

    opened by rowanc1 1
Owner
Executable Books
An open collaboration to create executable books with Jupyter
Executable Books
Get packages from a monorepo (pnpm, yarn, npm, lerna)

?? You can help the author become a full-time open-source maintainer by sponsoring him on GitHub. @egoist/get-packages Get packages from a monorepo (p

EGOIST 45 Jun 1, 2022
A monorepo for comma.ai web interfaces and packages

comma webapps This mono-repository contains the web applications and packages for the web UIs of comma.ai Contributing Just pick something and work on

null 5 Sep 27, 2022
Grupprojekt för kurserna 'Javascript med Ramverk' och 'Agil Utveckling'

JavaScript-med-Ramverk-Laboration-3 Grupprojektet för kurserna Javascript med Ramverk och Agil Utveckling. Utvecklingsguide För information om hur utv

Svante Jonsson IT-Högskolan 3 May 18, 2022
Hemsida för personer i Sverige som kan och vill erbjuda boende till människor på flykt

Getting Started with Create React App This project was bootstrapped with Create React App. Available Scripts In the project directory, you can run: np

null 4 May 3, 2022
Kurs-repo för kursen Webbserver och Databaser

Webbserver och databaser This repository is meant for CME students to access exercises and codealongs that happen throughout the course. I hope you wi

null 14 Jan 3, 2023
Script to fetch all NFT owners using moralis API. This script output is a data.txt file containing all owner addresses of a given NFT and their balances.

?? Moralis NFT Snapshot Moralis NFT API will only return 500 itens at a time when its is called. For that reason, a simple logic is needed to fetch al

Phill Menezes 6 Jun 23, 2022
Blogkit - A unified blogging engine built with Next.js

Blogkit (beta) Blogkit is a unified blog engine inspired by Sairin. Get started with starter templates Template Description blogkit-notion-starter Not

2nthony 7 Jun 9, 2022
Utilities for parsing and manipulating LaTeX ASTs with the Unified.js framework

unified-latex Monorepo for @unified-latex packages. These packages provide a JS/TypeScript interface for creating, manipulating, and printing LaTeX Ab

Jason Siefken 29 Dec 27, 2022
A unified and lightweight web application framework for multiple platforms.

Handlers.js Handlers.js is a unified and lightweight web application framework for multiple platforms. import handlerJS from "handlers.js"; const App

186526 7 Jul 26, 2022
Unified JavaScript logging system. KISS, light and library free.

Logger.js Logger.js is a JavaScript ES6 module that offers a unified console output across Firefox and Chromium based browsers. It handles standard er

Arthur Beaulieu 1 Oct 1, 2020
Incredible drastically simplifies creation of developer video content. It offers a unified workflow to storyboard, record, collaborate and produce the video.

?? Introduction Incredible drastically simplifies creation of developer video content. It offers a unified workflow to storyboard, record, collaborate

Incredible 113 Dec 6, 2022
A module containing utilities for osu!droid and osu! in general.

About This module is published for my own convenience so that I can use it across multiple projects of mine without having to manually transfer files,

Rian 10 Dec 23, 2022
⚡️ Monorepository containing all the source code for the Foxxie Project

⚡️ Monorepository containing all the source code for the Foxxie Project

Foxxie 5 Jun 30, 2022
Master Collection NFT. Mints NFTs on Ethereum containing unique combination of titles for fun.

Master NFT Collection Master NFT Collection is an NFT minting platform that mints NFTs that contain a unique combination of snazzy titles just for fun

MJ LEE 2 Mar 22, 2022
🧩 TypeScript utility type in order to ensure to return only properties (not methods) containing values in primitive types such as number or boolean (not Value Objects)

?? TypeScript Primitives type TypeScript utility type in order to ensure to return only properties (not methods) containing values in primitive types

CodelyTV 82 Dec 7, 2022
Pim 4 Jun 21, 2022
🦠🔬 Forta agent that detect deployment of smart contracts containing an exploit function

Attack Simulation Bot Description The agent detects deployment of smart contracts containing an exploit function. Using a simulation-based approach, t

Artem Kovalchuk 29 Dec 26, 2022
A frontend framework containing of tools for fast development of dashboard, panel, etc.

Khaos-Admin A frontend framework containing of tools for fast development of dashboard, panel, etc. Tools We Provide Fast Development: We handle data

Hamrah Mechanic 7 Nov 27, 2022
A template containing everything you need for creating your own Obsidian theme.

This is a sample theme for Obsidian (https://obsidian.md). First Time publishing a theme? Quick start First, choose Use this template. That will creat

Obsidian.md 15 Dec 28, 2022