@stencila/encoda
Codecs for executable document formats
Last updated 8 days ago by stencila-ci .
Apache-2.0 · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install @stencila/encoda 
SYNC missed versions from official npm registry.

Encoda

Codecs for structured, semantic, composable, and executable documents

Build status Code coverage NPM Contributors Docs Chat

Introduction

"A codec is a device or computer program for encoding or decoding a digital data stream or signal. Codec is a portmanteau of coder-decoder. - Wikipedia

Encoda provides a collection of codecs for converting between, and composing together, documents in various formats. The aim is not to achieve perfect lossless conversion between alternative document formats; there are already several tools for that. Instead the focus of Encoda is to use existing tools to encode and compose semantic documents in alternative formats.

Formats

Format Codec Approach Status Issues Coverage
Text
Plain text txt None β
Markdown md Extens α
LaTex latex - α
Microsoft Word docx rPNG α
Google Docs gdoc rPNG α
Open Document Text odt rPNG α
HTML html Extens α
JATS XML jats Extens α
JATS XML (Pandoc-based) jats-pandoc Extens α
Portable Document Format pdf rPNG α
Notebooks
Jupyter ipynb Native α
RMarkdown xmd Native α
Presentations
Microsoft Powerpoint pptx rPNG
Demo Magic dmagic Native β
Spreadsheets
Microsoft Excel xlsx Formula β
Google Sheets gsheet Formula
Open Document Spreadsheet ods Formula β
Tabular data
CSV csv None β
CSVY csvy None
Tabular Data Package tdp None β
Collections
Document Archive dar Extens ω
Filesystem Directory dir Extens ω
Data interchange, other
JSON json Native
JSON-LD [json-ld] Native [⚠][jsonld-issues]
JSON5 json5 Native
YAML yaml Native
Pandoc pandoc Native β
Reproducible PNG rpng Native β
XML xml Native
Transport
HTTP http

Key

<summary> Approach...</summary> How executable nodes (e.g. `CodeChunk` and `CodeExpr` nodes) are represented
  • Native: the format natively supports executable nodes
  • Extens.: executable nodes are supported via extensions to the format
  • rPNG: executable nodes are supported via reproducible PNG images
  • Formula: executable CodeExpr nodes are represented using formulae
<summary> Status...</summary>
  • ✗: Not yet implemented
  • ω: Work in progress
  • α: Alpha, initial implementation
  • β: Beta, ready for user testing
  • ✔: Ready for production use
<summary> Issues...</summary> Link to open issues and PRs for the format (please check there before submitting a new issue ????)

If you'd like to see a converter for your favorite format, look at the listed issues and comment under the relevant one. If there is no issue regarding the converter you need, create one.

Publishers

Several of the codecs in Encoda, deal with fetching content from a particular publisher. For example, to get an eLife article and read it in Markdown:

stencila convert https://elifesciences.org/articles/45187v2 ye-et-al-2019.md

Some of these publisher codecs deal with meta data. e.g.

stencila convert "Watson and Crick 1953" - --from crossref --to yaml
type: Article
title: Genetical Implications of the Structure of Deoxyribonucleic Acid
authors:
  - familyNames:
      - WATSON
    givenNames:
      - J. D.
    type: Person
  - familyNames:
      - CRICK
    givenNames:
      - F. H. C.
    type: Person
datePublished: '1953,5'
isPartOf:
  issueNumber: '4361'
  isPartOf:
    volumeNumber: '171'
    isPartOf:
      title: Nature
      type: Periodical
    type: PublicationVolume
  type: PublicationIssue
Source Codec Base codec/s Status Issues Coverage
General
HTTP http Based on Content-Type or extension β
Person
ORCID [orcid] jsonld β [⚠][orcid-issues] ![][orcid-cov]
Article metadata
DOI [doi] csl β [⚠][doi-issues] ![][doi-cov]
Crossref [crossref] jsonld β [⚠][crossref-issues] ![][crossref-cov]
Article content
eLife [elife] jats β [⚠][elife-issues] ![][elife-cov]
PLoS [plos] jats β [⚠][plos-issues] ![][plos-cov]

Install

The easiest way to use Encoda is to install the stencila command line tool. Encoda powers stencila convert, and other commands, in that CLI. However, the version of Encoda in stencila, can lag behind the version in this repo. So if you want the latest functionality, install Encoda as a Node.js package:

npm install @stencila/encoda --global

Use

Encoda is intended to be used primarily as a library for other applications. However, it comes with a simple command line script which allows you to use the convert function directly.

Converting files

encoda convert notebook.ipynb notebook.docx

Encoda will determine the input and output formats based on the file extensions. You can override these using the --from and --to options. e.g.

encoda convert notebook.ipynb notebook.xml --to jats

You can also convert to more than one file / format (in this case the --to argument only applies to the first output file) e.g.

encoda convert report.docx report.Rmd report.html report.jats

Converting folders

You can decode an entire directory into a Collection. Encoda will traverse the directory, including subdirectories, decoding each file matching your glob pattern. You can then encode the Collection using the dir codec into a tree of HTML files e.g.

encoda convert myproject myproject-published --to dir --pattern '**/*.{rmd, csv}'

Converting command line input

You can also read content from the first argument. In that case, you'll need to specifying the --from format e.g.

encoda convert "{type: 'Paragraph', content: ['Hello world!']}" --from json5 paragraph.md

You can send output to the console by using - as the second argument and specifying the --to format e.g.

encoda convert paragraph.md - --to yaml

Creating zip archives

Use the --zip option to create a Zip archive with the outputs of conversion. With --zip=yes a zip archive will always be created. With --zip=maybe, a zip archive will be created if there are more than two output files. This can be useful for formats such as HTML and Markdown, for which images and other media are stored in a sibling folder.

Option Description
--from The format of the input content e.g. --from md
--to The format for the output content e.g. --to html
--theme The theme for the output (only applies to HTML, PDF and RPNG output) e.g. --theme eLife. Either a

Thema theme name or a path/URL to a directory containing a styles.css and a index.js file. | | --standalone | Generate a standalone document, not a fragment (default true) | | --bundle | Bundle all assets (e.g images, CSS and JS) into the document (default false) | | --debug | Print debugging information |

Using with Executa

Encoda exposes the decode and encode methods of the Executa API. Register Encoda so that it can be discovered by other executors on your machine,

npm run register

You can then use Encoda as a plugin for Executa that provides additional format conversion capabilities. For example, you can use the query REPL on a Markdown document:

npx executa query CHANGELOG.md --repl

You can then use the REPL to explore the structure of the document and do things like create summary documents from it. For example, lets say from some reason we wanted to create a short JATS XML file with the five most recent releases of this package:

jmp > %format jats
jmp > %dest latest-releases.jats.xml
jmp > {type: 'Article', content: content[? type==`Heading` && depth==`1`] | [1:5]}

Which creates the latest-major-releases.jats.xml file:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.1 20151215//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article">
    <front>
        <title-group>
            <article-title/>
        </title-group>
        <contrib-group/>
    </front>
    <body>
        <sec>
            <title>
                <ext-link ext-link-type="uri" xlink:href="https://github.com/stencila/encoda/compare/v0.79.0...v0.80.0">0.80.0</ext-link> (2019-09-30)
            </title>
        </sec>
...

You can query a document in any format supported by Encoda. As another example, lets' fetch a CSV file from Github and get the names of it's columns:

npx executa query https://gist.githubusercontent.com/jncraton/68beb88e6027d9321373/raw/381dcf8c0d4534d420d2488b9c60b1204c9f4363/starwars.csv --repl
???? INFO  encoda:http Fetching "https://gist.githubusercontent.com/jncraton/68beb88e6027d9321373/raw/381dcf8c0d4534d420d2488b9c60b1204c9f4363/starwars.csv"
jmp > columns[].name
[
  'SetID',
  'Number',
  'Variant',
  'Theme',
  'Subtheme',
  'Year',
  'Name',
  'Minifigs',
  'Pieces',
  'UKPrice',
  'USPrice',
  'CAPrice',
  'EUPrice',
  'ImageURL',
  'Owned',
  'Wanted',
  'QtyOwned',
]
jmp >

See the %help REPL command for more examples.

Note: If you have executa installed globally, then the npx prefix above is not necessary.

Documentation

Self-hoisted (documentation converted from various formats to html) and API documentation (generated from source code) is available at: https://stencila.github.io/encoda.

Develop

Check how to contribute back to the project. All PRs are most welcome! Thank you!

Clone the repository and install a development environment:

git clone https://github.com/stencila/encoda.git
cd encoda
npm install

You can manually test conversion using the ts-node and the cli.ts script:

npm run cli -- convert simple.md simple.html

There is a bash script to make that a little shorter and more like real life usage:

./encoda convert simple.md simple.html

If that is a bit slow, compile the Typescript to Javascript first and use node directly:

npm run build
node dist/cli convert simple.md simple.html

If you are using VSCode, you can use the Auto Attach feature to attach to the CLI when running the cli:debug NPM script:

npm run cli:debug -- convert simple.gdoc simple.ipynb

Testing

Running tests locally

Run the test suite using:

npm test

Or, run a single test file e.g.

npx jest tests/xlsx.test.ts --watch

To display debug logs during testing set the environment variable DEBUG=1, e.g.

DEBUG=1 npm test

To get coverage statistics:

npm run cover

There's also a Makefile if you prefer to run tasks that way e.g.

make lint cover

Running test in Docker

You can also test this package using with a Docker container:

npm run test:docker

Writing tests

Recording and using network fixtures

As far as possible, tests should be able to run with no network access. We use Nock Back to record and play back network requests and responses. Use the nockRecord helper function for this with the convention of starting the fixture file with nock-record- e.g.

const stopRecording = await nockRecord('nock-record-<name-of-test>.json')
// Do some things that connect to the interwebs
stopRecording()

Note that the util/http module has caching so that you may need to remove the cache for the recording of fixtures to work e.g. rm -rf ~/.config/stencila/encoda/cache/

As part of continuous integration, tests are run with both NOCK_MODE=wild (which ignores recording) to test that tests still work when using real network connections, and NOCK_MODE=dryrun (for speed and consistency with the default).

Contribute

We ???? contributions! All contributions: ideas ????, examples ????, bug reports ????, documentation ????, code ????, questions ????. See CONTRIBUTING.md for more on where to start. You can also provide your feedback on the Community Forum and Gitter channel.

Contributors


Aleksandra Pawlik

???? ???? ????

Nokome Bentley

???? ???? ????

Jacqueline

???? ????

Hamish Mackenzie

???? ????

Alex Ketch

???? ???? ????

Ben Shaw

???? ????

Phil Neff

????

Raniere Silva

????

Lorenzo Cangiano

????

FAtherden-eLife

???? ????

Giorgio Sironi

????
<summary> Add a contributor...</summary>

To add youself, or someone else, to the above list, either,

  1. Ask the @all-contributors bot to do it for you by commenting on an issue or PR like this:

    @all-contributors please add @octocat for bugs, tests and code

  2. Use the all-contributors CLI to do it yourself:

    npx all-contributors add octocat bugs, tests, code
    

See the list of contribution types.

Acknowledgments

Encoda relies on many awesome opens source tools (see package.json for the complete list). We are grateful ❤ to their developers and contributors for all their time and energy. In particular, these tools do a lot of the heavy lifting ???? under the hood.

Tool Use
Ajv Ajv is "the fastest JSON Schema validator for Node.js and browser". Ajv is not only fast, it also has an impressive breadth of functionality. We use Ajv for the validate() and coerce() functions to ensure that ingested data is valid against the Stencila schema.
Citation.js Citation.js converts bibliographic formats like BibTeX, BibJSON, DOI, and Wikidata to CSL-JSON. We use it to power the codecs for those formats and APIs.
Frictionless Data datapackage-js from the team at Frictionless Data is a Javascript library for working with Data Packages. It does a lot of the work in converting between Tabular Data Packages and Stencila Datatables.
Glitch Digital Glitch Digital's structured-data-testing-tool is a library and command line tool to help inspect and test for Structured Data. We use it to check that the HTML generated by Encoda can be read by bots ????
Pa11y Pa11y provides a range of free and open source tools to help designers and developers make their web pages more accessible. We use pa11y to test that HTML generated produced by Encoda meets the Web Content Accessibility Guidelines (WCAG) and Axe rule set.
Pandoc Pandoc is a "universal document converter". It's able to convert between an impressive number of formats for textual documents. Our Typescript definitions for Pandoc's AST allow us to leverage this functionality from within Node.js while maintaining type safety. Pandoc powers our converters for Word, JATS and Latex. We have contributed to Pandoc, including developing its JATS reader.
Puppeteer Puppeteer is a Node library which provides a high-level API to control Chrome. We use it to take screenshots of HTML snippets as part of generating rPNGs and we plan to use it for generating PDFs.
Remark Remark is an ecosystem of plugins for processing Markdown. It's part of the unified framework for processing text with syntax trees - a similar approach to Pandoc but in Javascript. We use Remark as our Markdown parser because of it's extensibility.
SheetJs SheetJs is a Javascript library for parsing and writing various spreadsheet formats. We use their community edition to power converters for CSV, Excel, and Open Document Spreadsheet formats. They also have a pro version if you need extra support and functionality.

Many thanks ❤ to the Alfred P. Sloan Foundation and eLife for funding development of this tool.

Current Tags

  • 0.94.0                                ...           latest (8 days ago)
  • 0.55.1-alpha                                ...           next (a year ago)

146 Versions

  • 0.94.0                                ...           8 days ago
  • 0.93.14                                ...           10 days ago
  • 0.93.13                                ...           10 days ago
  • 0.93.12                                ...           10 days ago
  • 0.93.11                                ...           15 days ago
  • 0.93.10                                ...           18 days ago
  • 0.93.9                                ...           19 days ago
  • 0.93.8                                ...           a month ago
  • 0.93.7                                ...           2 months ago
  • 0.93.6                                ...           2 months ago
  • 0.93.5                                ...           2 months ago
  • 0.93.4                                ...           2 months ago
  • 0.93.3                                ...           2 months ago
  • 0.93.2                                ...           2 months ago
  • 0.93.1                                ...           2 months ago
  • 0.93.0                                ...           2 months ago
  • 0.92.0                                ...           2 months ago
  • 0.91.0                                ...           3 months ago
  • 0.90.3                                ...           3 months ago
  • 0.90.2                                ...           3 months ago
  • 0.90.1                                ...           3 months ago
  • 0.90.0                                ...           3 months ago
  • 0.89.0                                ...           3 months ago
  • 0.88.2                                ...           3 months ago
  • 0.88.1                                ...           3 months ago
  • 0.88.0                                ...           3 months ago
  • 0.87.0                                ...           3 months ago
  • 0.86.2                                ...           3 months ago
  • 0.86.1                                ...           3 months ago
  • 0.86.0                                ...           4 months ago
  • 0.85.3                                ...           4 months ago
  • 0.85.2                                ...           4 months ago
  • 0.85.1                                ...           4 months ago
  • 0.85.0                                ...           4 months ago
  • 0.84.0                                ...           4 months ago
  • 0.83.2                                ...           4 months ago
  • 0.83.1                                ...           4 months ago
  • 0.83.0                                ...           5 months ago
  • 0.81.3                                ...           6 months ago
  • 0.81.2                                ...           7 months ago
  • 0.81.1                                ...           7 months ago
  • 0.81.0                                ...           8 months ago
  • 0.80.2                                ...           8 months ago
  • 0.80.1                                ...           8 months ago
  • 0.80.0                                ...           8 months ago
  • 0.79.0                                ...           8 months ago
  • 0.78.2                                ...           8 months ago
  • 0.78.1                                ...           8 months ago
  • 0.78.0                                ...           9 months ago
  • 0.77.1                                ...           9 months ago
  • 0.77.0                                ...           9 months ago
  • 0.76.0                                ...           9 months ago
  • 0.75.4                                ...           9 months ago
  • 0.75.3                                ...           9 months ago
  • 0.75.2                                ...           9 months ago
  • 0.75.1                                ...           9 months ago
  • 0.75.0                                ...           9 months ago
  • 0.74.0                                ...           9 months ago
  • 0.73.0                                ...           9 months ago
  • 0.72.0                                ...           9 months ago
  • 0.71.3                                ...           9 months ago
  • 0.71.2                                ...           9 months ago
  • 0.71.1                                ...           9 months ago
  • 0.71.0                                ...           9 months ago
  • 0.70.0                                ...           9 months ago
  • 0.69.0                                ...           9 months ago
  • 0.68.0                                ...           9 months ago
  • 0.67.2                                ...           9 months ago
  • 0.67.1                                ...           9 months ago
  • 0.67.0                                ...           9 months ago
  • 0.66.0                                ...           9 months ago
  • 0.65.1                                ...           9 months ago
  • 0.65.0                                ...           10 months ago
  • 0.64.0                                ...           10 months ago
  • 0.63.0                                ...           10 months ago
  • 0.62.7                                ...           10 months ago
  • 0.62.6                                ...           10 months ago
  • 0.62.5                                ...           10 months ago
  • 0.62.4                                ...           10 months ago
  • 0.62.3                                ...           10 months ago
  • 0.62.2                                ...           10 months ago
  • 0.62.1                                ...           10 months ago
  • 0.62.0                                ...           10 months ago
  • 0.61.2                                ...           10 months ago
  • 0.61.1                                ...           10 months ago
  • 0.61.0                                ...           10 months ago
  • 0.60.4                                ...           10 months ago
  • 0.60.3                                ...           10 months ago
  • 0.60.2                                ...           10 months ago
  • 0.60.1                                ...           10 months ago
  • 0.60.0                                ...           10 months ago
  • 0.59.2                                ...           10 months ago
  • 0.59.1                                ...           a year ago
  • 0.59.0                                ...           a year ago
  • 0.58.0                                ...           a year ago
  • 0.57.0                                ...           a year ago
  • 0.56.0                                ...           a year ago
  • 0.55.1                                ...           a year ago
  • 0.55.1-alpha                                ...           a year ago
  • 0.55.0                                ...           a year ago
  • 0.54.0                                ...           a year ago
  • 0.53.7                                ...           a year ago
  • 0.53.6                                ...           a year ago
  • 0.53.5                                ...           a year ago
  • 0.53.4                                ...           a year ago
  • 0.53.3                                ...           a year ago
  • 0.53.2                                ...           a year ago
  • 0.53.1                                ...           a year ago
  • 0.53.0                                ...           a year ago
  • 0.52.1                                ...           a year ago
  • 0.52.0                                ...           a year ago
  • 0.51.2                                ...           a year ago
  • 0.51.1                                ...           a year ago
  • 0.51.0                                ...           a year ago
  • 0.50.4                                ...           a year ago
  • 0.50.3                                ...           a year ago
  • 0.50.2                                ...           a year ago
  • 0.50.1                                ...           a year ago
  • 0.50.0                                ...           a year ago
  • 0.49.2                                ...           a year ago
  • 0.49.1                                ...           a year ago
  • 0.49.0                                ...           a year ago
  • 0.48.0                                ...           a year ago
  • 0.47.1                                ...           a year ago
  • 0.47.0                                ...           a year ago
  • 0.46.3                                ...           a year ago
  • 0.46.2                                ...           a year ago
  • 0.46.1                                ...           a year ago
  • 0.46.0                                ...           a year ago
  • 0.45.0                                ...           a year ago
  • 0.44.2                                ...           a year ago
  • 0.44.1                                ...           a year ago
  • 0.44.0                                ...           a year ago
  • 0.43.4                                ...           a year ago
  • 0.43.3                                ...           a year ago
  • 0.43.2                                ...           a year ago
  • 0.43.1                                ...           a year ago
  • 0.43.0                                ...           a year ago
  • 0.42.0                                ...           a year ago
  • 0.41.0                                ...           a year ago
  • 0.40.1                                ...           a year ago
  • 0.40.0                                ...           a year ago
  • 0.39.0                                ...           a year ago
  • 0.38.1                                ...           a year ago
  • 0.38.0                                ...           a year ago
  • 0.37.0                                ...           a year ago
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 96
Last Month 244
Dependencies (60)
Dependents (2)

Copyright 2014 - 2016 © taobao.org |