Description

New attempt at custom Tokens, Renderers for Marked.js. Faster than #1872.

May be a possible fix for: #1373, #1693, #1695, #2061,

Users can add a custom Tokenizer and Renderer which will be executed without needing to overwrite existing components, with the extension in the following format:

const myExtension = {
  extensions: [{
    name: 'underline',
    start(src) { return src.match(/:/)?.index; },  // First characters of your token so Marked.js knows to stop and check for a match
    level: 'block',        // Is this a block-level or inline-level tokenizer?
    tokenizer(src, tokens) {
      const rule = /^:([^\n]*)(?:\n|$)/;  // Regex for the complete token
      const match = rule.exec(src);
      if (match) {
        return {                          // Token to generate
          type: 'underline',                // Should match "name" above
          raw: match[0],                    // The text that you want your token to consume from the source
          text: match[1].trim()             // Any custom properties you want the Renderer to access
        };
      }
    },
    renderer (token) {
      return `<u>${token.text}</u>\n`;
    }]
};

The extension(s) can then be loaded like so:

marked.use(myExtension, extension2, extension3);

\\ EQUIVALENT TO:

marked.use(myExtension);
marked.use(extension2);
marked.use(extension3);

Benchmarks (on my laptop):

Master from Feb 2021

es5 marked completed in 4260ms and passed 82.74%
es6 marked completed in 4289ms and passed 82.74%
es5 marked (gfm) completed in 4518ms and passed 82.13%
es6 marked (gfm) completed in 4504ms and passed 82.13%
es5 marked (pedantic) completed in 4596ms and passed 61.48%
es6 marked (pedantic) completed in 4783ms and passed 61.48%
commonmark completed in 3617ms and passed 100.00%
markdown-it completed in 3646ms and passed 89.21%

Master (current)

es5 marked completed in 4504ms and passed 86.90%
es6 marked completed in 4622ms and passed 86.90%
es5 marked (gfm) completed in 4776ms and passed 86.29%
es6 marked (gfm) completed in 5029ms and passed 86.29%
es5 marked (pedantic) completed in 4674ms and passed 71.03%
es6 marked (pedantic) completed in 5007ms and passed 71.03%
commonmark completed in 3678ms and passed 100.00%
markdown-it completed in 3672ms and passed 89.21%

This PR

es5 marked completed in 4761ms and passed 86.90%
es6 marked completed in 4814ms and passed 86.90%
es5 marked (gfm) completed in 5115ms and passed 86.29%
es6 marked (gfm) completed in 5105ms and passed 86.29%
es5 marked (pedantic) completed in 4972ms and passed 71.03%
es6 marked (pedantic) completed in 5069ms and passed 71.03%
commonmark completed in 3619ms and passed 100.00%
markdown-it completed in 3735ms and passed 89.21%

This PR with all extensions running at the top of the Lexer rather than using "before" to spread them out:

es5 marked completed in 4591ms and passed 86.90%
es6 marked completed in 4644ms and passed 86.90%
es5 marked (gfm) completed in 4891ms and passed 86.29%
es6 marked (gfm) completed in 4804ms and passed 86.29%
es5 marked (pedantic) completed in 4770ms and passed 71.03%
es6 marked (pedantic) completed in 4799ms and passed 71.03%
commonmark completed in 3617ms and passed 100.00%
markdown-it completed in 3587ms and passed 89.21%

Contributor

• [x] Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
• [ ] no tests required for this PR.
• [x] If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• [ ] CI is green (no forced merge required).
• [ ] Squash and Merge PR following conventional commit guidelines.

You can customize the result with a customized renderer.

var renderer = new marked.Renderer()

renderer.header = function(text, level) {
  return '<div class="h-' + level + '">' + text + '</div>'
}


var parse = function(src, options) {
  options = options || {};
  return marked.parser(marked.lexer(src, options), options, renderer);
}

console.log(parse('#h1'))

The renderer API:

blockcode: function(code, lang)
blockquote: function(text)
blockhtml: function(html)

header: function(text, level)
paragraph: function(text)

hrule: function()

list: function(contents, isOrdered)
listitem: function(text)

table: function(header, body)
tablerow: function(content)
tablecell: function(text, flags)
// flags: {header: false, align: 'center'}

I am doing my best to make the renderer API comptable with sundown's renderer.

This is the feature I really need. If marked has, I can drop the c binding sundown of robotskirt.

Description

An attempt at #1695. Not sure about speed or elegance here, but wanted some feedback to see if this is even a reasonable route to take. If so, I would appreciate some troubleshooting to get this cleaner and to fix the issue with the broken test case.

Ideally, this would allow users to extend the Lexer by plugging in custom tokenizers at a chosen space in the lexer pipeline, and the params object exposes all of the required parameters to make functions with different signatures work.

Notes:

• I only included the block tokenizers here to start.
• This seems to break something with the def tokenizer. I don't fully understand what, but it makes Commonmark #187 fail.

Contributor

• [ ] Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
• [ ] no tests required for this PR.
• [ ] If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• [ ] CI is green (no forced merge required).
• [ ] Squash and Merge PR following conventional commit guidelines.

With the sanitize option on it is possible to create a link with a javascript: protocol with the following: [URL](javascript&#58document;alert(1)).

HTML entities in the browser are not strict and parse what they can and leaving the rest behind. For example &#xNNanything; would parse the NN hex values but leave behind the string "anything;".

"javascript&#58document;" with the regex /&([#\w]+);/ returns "58document" and is parsed by String.fromCharCode to "". Because of this the later tests only sees the javascript keyword without the :. However the browser parses this to: "javascript:document;".

Add support for Github Task Lists under the gfm flag.

Changes to API

• list(string body, boolean ordered, boolean taskList)
• listitem(string text, [boolean checked]).

checked is defined when you have a list item which starts with [ ] or [x].If defined its a boolean depending on whether the x is present. When checked is defined we add a input type type checkbox to the list item and add the class task-list-item-checkbox.

taskList is true if a list has any list items where checked is defined. When true we add the class task-list to the list.

Resolves chjj/marked#107

I tested locally in the project I wanted to use this with:

• by default it behaves the as it does now: links open in the same tab/window
• setting it to true via setOptions: links opens in a new tab/window
• setting it to false via setOptions: links open in the same tab/window

Also, checked the README entry by looking at it on github.

Unit tests would be nice but I don't have time to grok how the tests work in this repo and other stuff is also not unit tested (e.g. smartLists)

Thanks for the excellent md parser!

There are a bunch of various extensions to markdown eg http://freewisdom.org/projects/python-markdown/Extra http://michelf.com/projects/php-markdown/extra/ http://maruku.rubyforge.org/proposal.html

While not all are exactly critical, things like super/subscript, tables, definition lists, abbreviations can go a long way to make writing md documents easier.

Also simple transformations (like texttile http://rpc.textpattern.com/help/?item=intro) can make the text much more readable For example ... -> … ' ' -> ‘ ’ " " -> “ ” -- -> – (c) -> © (r) -> ® and so on .

Ideally the goal should be to never write pure html for any typographical feature.

Marked version: 4.0.0

Describe the bug I updated marked in our project from ^3.0.2 to 4.0.0 (before it worked fine), and after the update I get an error marked__WEBPACK_IMPORTED_MODULE_5___default(...) is not a function when using it. It seems that the module is no longer exported properly, I tried to change the import import { marked } from 'marked' as well as using marked.parse() as shown in the readme, but that doesn't seem to help.

To Reproduce Steps to reproduce the behavior: Use a webpack/npm project, import by means of import marked from 'marked'. Use the marked() function.

Description

1. Decided to go with preversion instead of anything related t publish...version only update the package.json file, which needs to be committed anyway; so, thinking anything that needs to be updated and part of a commit should be able to tag along.
2. Created RELEASE.md for instructions on building a release.

This gives us a three step process when we don't include the setup things - and makes us completely within NPM commands:

1. npm test
2. npm version [major|minor|patch]
3. npm publish

Replaces #1064

Review

Submitter

• [ ] All tests pass (CI should take care of this, once in place). One known failing test.
• [ ] All lint checks pass (CI should take care of this, once in place). Not in place yet.
• Tests
  • [ ] Test(s) exist to ensure functionality works (if no new tests added, list which tests cover this functionality).
  • [x] No tests required for this PR.
• [ ] Is release:
  • [ ] Version in package.json has been updated (see RELEASE.md).
  • [ ] The marked.min.js has been updated; or,
  • [ ] release does not change library.

Reviewer

??

Based on the conversation in https://github.com/markedjs/marked/pull/2112#discussion_r656741740, this is the start of an attempt to have the Tokenizers handle the lexing of their own children tokens, rather than making the Lexer.js do it.

For block tokens this was relatively simple. For inline tokens it's also not a huge issue, except for the ugliness that comes with passing in inRawBlock and inLink to a bunch of the Tokenizers since it kind of muddies up the legibility in what the Tokenizers are actually doing. Passing those values around seems like a code smell we could avoid but I don't know how, or how much those variables actually need to be passed around. Any thoughts? I wanted to get some early feedback before going through the whole thing.

Edit : What about refactoring the inLink and inRawBlock flags to instead be properties of the Lexer? I.e. in the constructor:

lexer.state = { //or lexer.flags, etc...
  inLink : false,
  inRawBlock : false
}

And a second question: do we want the logic of inline() from Lexer.js to also be handled by the Tokenizers themselves?

Contributor

• [ ] Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
• [ ] no tests required for this PR.
• [ ] If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• [ ] CI is green (no forced merge required).
• [ ] Squash and Merge PR following conventional commit guidelines.

There is a rising demand to move towards modularization of *marked * and separate different components in different files(grammar-rules, lexers, parsers and renderers). This though not have been mentioned explicitly, is evident from issues #743, #717, #704.

A properly modularized code would make maintenance easy and also allow addition of new grammar, implementation of custom parser, renderer and would address separation of concerns properly.

Like may be we can have an /src folder to fiddle with the code in development and use /lib for deliverable file.

In the known extensions docs, there's a list of extensions and an instruction to use them via marked.use(extension). However, I looked at the source of some of them, and it looks like they do not export a Marked extension but a Tokenizer or Renderer extension. This is also how it's stated to be done in the custom extensions docs.

Therefore I think the instruction should be to use them as marked.use({ extensions: [/* ... */] }) instead.

BTW I find it very confusing that there's this naming ambiguity for the word "extension". The differentiation is vague in the docs, the only thing I could find about the difference of marked.use({ renderer }) vs marked.use({ extensions: [{ renderer }] }) is that the latter runs the extension renderer as a preprocessing, whereas the former one merges with the existing renderer (?)

Would be nice if the extensions API gets a bit more straightforward (:

Hi,

please find a proof of concept in this PR.

I think it'd be better to just remove man/marked.1* from the git repository and add them to .gitignore, since now they are generated from marked-man.

Describe the feature Currently renderer methods get arguments cooked by marked ~Renderer~ Parser, as in code(code, infostring, escaped). On the other hand, extensions rendered methods get a token parameter with all its properties. I request that all renderer methods get a token.

Why is this feature necessary? For example this.renderer.list won't know if the list is loose.

Describe alternatives you've considered I could have asked for this.renderer.list to get a (body, ordered, start, loose) signature. But passing the token around systematically seems to be much simpler.

Marked version:

• v4.0.18

Describe the bug A clear and concise description of what the bug is.

Copy from https://github.com/volca/markdown-preview/issues/135.

The case below, it does not parse syntax correctly.

% cat test.md
* ×: あれ、**`foo`これ**、それ
* ○: あれ、 **`foo`これ**、それ
* ×: あれ、**`foo`これ** 、それ

* ○: あれ、**fooこれ**、それ
* ○: あれ、 **fooこれ**、それ
* ○: あれ、**fooこれ** 、それ

% npx marked --version
4.0.18

% npx marked < test.md
<ul>
<li><p>×: あれ、**<code>foo</code>これ**、それ</p>
</li>
<li><p>○: あれ、 <strong><code>foo</code>これ</strong>、それ</p>
</li>
<li><p>×: あれ、**<code>foo</code>これ** 、それ</p>
</li>
<li><p>○: あれ、<strong>fooこれ</strong>、それ</p>
</li>
<li><p>○: あれ、 <strong>fooこれ</strong>、それ</p>
</li>
<li><p>○: あれ、<strong>fooこれ</strong> 、それ</p>
</li>
</ul>

With Japanese punctuation (、), strong syntax (**), and code syntax (`), it needs some space to make them parsed correctly (The former 3 examples).

Although, without code syntax, no extra space is required (The latter 3 examples).

So it isn't a syntax parsing problem with CJK symbol characters?

To Reproduce Steps to reproduce the behavior:

As above.

Expected behavior A clear and concise description of what you expected to happen.

Parse the syntax correctly as Pandoc.

% pandoc --version
pandoc.exe 2.18
Compiled with pandoc-types 1.22.2, texmath 0.12.5, skylighting 0.12.3,
citeproc 0.7, ipynb 0.2, hslua 2.2.0
Scripting engine: Lua 5.4
User data directory: C:\Users\yasuda\AppData\Roaming\pandoc
Copyright (C) 2006-2022 John MacFarlane. Web:  https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

% pandoc < test.md
<ul>
<li><p>×: あれ、<strong><code>foo</code>これ</strong>、それ</p></li>
<li><p>○: あれ、 <strong><code>foo</code>これ</strong>、それ</p></li>
<li><p>×: あれ、<strong><code>foo</code>これ</strong> 、それ</p></li>
<li><p>○: あれ、<strong>fooこれ</strong>、それ</p></li>
<li><p>○: あれ、 <strong>fooこれ</strong>、それ</p></li>
<li><p>○: あれ、<strong>fooこれ</strong> 、それ</p></li>
</ul>

What pain point are you perceiving? Since MarkedJS's inception most major versions have increased time needed to complete parsing. The numbers shown below were captured on second page load /w 6x CPU slowdown - only Marked version being revised, nothing else. Marked processing time has increased more than 2x since v0.8.2

• Version 0.8.2

• Version 2.0.3

• Version 4.0.9

Describe the solution you'd like Perhaps it's time to look at an optimization pass and potential short circuits that could be taken to restore some previously lost performance? I feel performance should remain somewhat consistent...

Like https://cdn.jsdelivr.net/gh/markedjs/marked/marked.min.js content header:

/**
 * marked - a markdown parser
 * Copyright (c) 2011-2021, Christopher Jeffrey. (MIT Licensed)
 * https://github.com/markedjs/marked
 */

I don't know current version and I don't know how to verify. I think should add version like that:

/**
 * marked 3.0.0 - a markdown parser
 * Copyright (c) 2011-2021, Christopher Jeffrey. (MIT Licensed)
 * https://github.com/markedjs/marked
 */