logo

idio filter

← Back to Filter List

idio


↓ examples

Splits files into sections based on comments like ### "foo" Replacement for idiopidae. Should be fully backwards-compatible. For more information about the settings starting with ply-, see the PLY YACC parser documentation http://www.dabeaz.com/ply/ply.html#ply_nn36

Aliases for this filter

  • idio
  • id
  • idiopidae
  • htmlsections

Converts from file formats:

  • .*

To file formats:

  • .html
  • .tex
  • .svg
  • .txt
  • .png
  • .bmp
  • .gif
  • .jpg

Available settings:

SettingDescriptionDefault
add-new-filesBoolean or list of extensions/patterns to match.False
added-in-versionDexy version when this filter was first available.
additional-doc-filtersFilters to apply to additional documents created as side effects.{}
additional-doc-settingsSettings to apply to additional documents created as side effects.{}
allow-unknown-extWhether to allow unknown file extensions to be parsed with the TextLexer by default instead of raising an exception.True
allow-unprintable-inputWhether to allow unprintable input to be replaced with dummy text instead of raising an exception.True
data-typeAlias of custom data class to use to store filter output.generic
examplesTemplates which should be used as examples for this filter.['idio', 'htmlsections']
exclude-add-new-filesList of patterns to skip even if they match add-new-files.[]
exclude-new-files-from-dirList of directories to skip when adding new files.[]
extFile extension to output.None
extension-mapDictionary mapping input extensions to default output extensions.None
formatter-settingsList of all settings which will be passed to the formatter constructor.['style', 'full', 'linenos', 'noclasses']
fullPygments formatter option: output a 'full' document including header/footer tags.None
helpHelpstring for plugin.Splits files into sections based on comments like ### "foo" Replacement for idiopidae. Should be fully backwards-compatible. For more information about the settings starting with ply-, see the PLY YACC parser documentation http://www.dabeaz.com/ply/ply.html#ply_nn36
highlightWhether to apply syntax highlighting to sectional output.None
input-extensionsList of extensions which this filter can accept as input.['.*']
keep-originalsWhether, if additional-doc-filters are specified, the original unmodified docs should also be added.False
lexerThe name of the pygments lexer to use (will normally be determined automatically, only use this if you need to override the default setting or your filename isn't mapped to the lexer you want to use.None
lexer-argsDictionary of custom arguments to be passed directly to the lexer.{}
lexer-settingsList of all settings which will be passed to the lexer constructor.[]
line-numbersAlternative name for 'linenos'.None
linenosWhether to include line numbers. May be set to 'table' or 'inline'.None
mkdirA directory which should be created in working dir.None
mkdirsA list of directories which should be created in working dir.[]
noclassesIf set to true, token tags will not use CSS classes, but inline styles.None
nodocWhether filter should be excluded from documentation.False
outputWhether to output results of this filter by default by reporters such as 'output' or 'website'.False
output-extensionsList of extensions which this filter can produce as output.['.html', '.tex', '.svg', '.txt', '.png', '.bmp', '.gif', '.jpg']
override-workspace-exclude-filtersIf True, document will be populated to other workspaces ignoring workspace-exclude-filters.False
ply-lextabName of lexer tabfile (.py extension will be added) to be stored in ply-outputdir if ply-optimize is set to 1.id_lextab
ply-optimizeWhether to use optimized mode for the lexer.1
ply-outputdirLocation relative to where you run dexy in which ply will store table files. Defaults to dexy's log directory.None
ply-parsetabName of parser tabfile (.py extension will be added) to be stored in ply-outputdir if ply-write-tables set to 1.id_parsetab
ply-write-tablesWhether to generate parser table files (which will be stored in ply-outputdir and named ply-tabmodule).1
preserve-prior-data-classWhether output data class should be set to match the input data class.False
remove-leadingIf a document starts with empty section named '1', remove it.False
require-outputShould dexy raise an exception if no output is produced by this filter?True
skip-extensionsBecause |idio gets applied to *.*, need to make it easy to skip non-textual files..odt
styleFormatter style to output.default
tagsTags which describe the filter.[]
unprintable-input-textDummy text to use instead of unprintable binary input.not printable
variablesA dictionary of variable names and values to make available to this filter.{}
varsA dictionary of variable names and values to make available to this filter.{}
workspace-exclude-filtersFilters whose output should be excluded from workspace.['pyg']
workspace-includesIf set to a list of filenames or extensions, only these will be populated to working dir.None

Idio Example

The idio filter splits source code according to comments found in the code. Many different comment formats are supported, please let us know if you need additional formats supported for your programming language.

Shortcut Quoted Form

You can write idio comments by using three comment characters and putting the section name in quotes, like this:

### "foo"
x = 6

### "bar"
y = 7

The resulting section names are:

1, foo, bar

For languages which support // as a comment char, just use three slashes instead of two:

/// "include"
#include <stdio.h>

/// "main"
int main(int argc, char *argv[]) {
    puts("Hello world!");
    return 0;
}

Here are the section names:

1, include, main

Here is the script output:

Hello world!

(A /* */ multiline-style comment is also supported, see the next section.)

The ; comment character is supported for Clojure:

;;; "foo"
(println "hello")

;;; "bar"
(println "world")

Here is the output from running the full clojure script:

hello
world

Here is the REPL output from just the foo section of the clojure script:

user=> (println "hello")
hello
nil

The % comment character is supported, for LaTeX:

%%% "foo"
Some LaTeX.

%%% "bar"
Some more LaTeX.

Here are the section names:

1, foo, bar

Long Form

The original idio syntax uses the @export keyword followed by a name, which need not be in quotes. This format is still supported:

### @export foo
x = 6

### @export "bar"
y = 7

The resulting section names are:

1, foo, bar

To use a multi-line style comment, use three initial asterisks and the @export keyword:

/*** @export include */
#include <stdio.h>

/*** @export main */
int main(int argc, char *argv[]) {
    puts("Hello world!");
    return 0;
}

Here are the section names:

1, include, main

Here is the script output:

Hello world!

HTML style comments are also supported:

<!-- @export foo -->
foo

<!-- @export "bar" -->
bar

Here are the section names:

1, foo, bar

htmlsections Filter

Here is some HTML split into sections:

<html>
    <head>
    </head>
    <body>
        <!-- section "foo" -->
        <div style="border: thin solid blue; width: 100px;">
            <p>hello!</p>
        </div>
        <!-- section "bar" -->
        <div style="border: thick dotted red; width: 100px;">
            <p>how are you?</p>
        </div>
        <!-- section "end" -->
    </body>
</html>

Here is another HTML document which includes some sections:

<html>
    <body>
        Here is the 'foo' section:

        {{ d['sections.html|htmlsections']['foo'] }}

        Here is the 'bar' section:

        {{ d['sections.html|htmlsections']['bar'] }}
    </body>
</html>

After running dexy, the sections are included:

<html>
    <body>
        Here is the 'foo' section:

                <div style="border: thin solid blue; width: 100px;">
            <p>hello!</p>
        </div>


        Here is the 'bar' section:

                <div style="border: thick dotted red; width: 100px;">
            <p>how are you?</p>
        </div>

    </body>
</html>

Here is how we specify this:

filters:
    - docs.html|jinja:
        - sections.html|htmlsections
Filter Source Code
class Id(PygmentsFilter):
    """
    Splits files into sections based on comments like ### "foo"

    Replacement for idiopidae. Should be fully backwards-compatible.

    For more information about the settings starting with ply-, see the PLY
    YACC parser documentation http://www.dabeaz.com/ply/ply.html#ply_nn36
    """
    aliases = ['idio', 'id', 'idiopidae', 'htmlsections']
    _settings = {
            'examples' : ['idio', 'htmlsections'],
            'highlight' : ("Whether to apply syntax highlighting to sectional output.", None),
            'skip-extensions' : ("Because |idio gets applied to *.*, need to make it easy to skip non-textual files.", (".odt")),
            'remove-leading' : ("If a document starts with empty section named '1', remove it.", False),
            'ply-optimize' : ("Whether to use optimized mode for the lexer.", 1),
            'ply-write-tables' : ("Whether to generate parser table files (which will be stored in ply-outputdir and named ply-tabmodule).", 1),
            'ply-outputdir' : ("Location relative to where you run dexy in which ply will store table files. Defaults to dexy's log directory.", None),
            'ply-parsetab' : ("Name of parser tabfile (.py extension will be added) to be stored in ply-outputdir if ply-write-tables set to 1.", 'id_parsetab'),
            'ply-lextab' : ("Name of lexer tabfile (.py extension will be added) to be stored in ply-outputdir if ply-optimize is set to 1.", 'id_lextab'),
            'output-extensions' : PygmentsFilter.MARKUP_OUTPUT_EXTENSIONS + PygmentsFilter.IMAGE_OUTPUT_EXTENSIONS
            }

    def process(self):
        try:
            input_text = str(self.input_data)
        except UnicodeDecodeError:
            self.output_data['1'] = "non textual"
            self.output_data.save()
            return

        lexer.outputdir = self.setting('ply-outputdir')
        lexer.errorlog = self.doc.wrapper.log
        lexer.remove_leading = self.setting('remove-leading')
        parser.outputdir = self.setting('ply-outputdir')
        parser.errorlog = self.doc.wrapper.log
        parser.write_tables = self.setting('ply-write-tables')

        _lexer = lexer.clone()
        _lexer.sections = []
        _lexer.level = 0
        start_new_section(_lexer, 0, 0, _lexer.level)

        parser.parse(input_text + "\n", lexer=_lexer)
        strip_trailing_newline(_lexer)
        parser_output = _lexer.sections

        pyg_lexer = self.create_lexer_instance()
        pyg_formatter = self.create_formatter_instance()

        # TODO fix file extension if highlight is set to false
        do_highlight = self.setting('highlight')
        if do_highlight is None:
            if self.alias in ('htmlsections',):
                do_highlight = False
                if self.output_data.setting('canonical-output') is None:
                    self.output_data.update_settings({'canonical-output' : True})
            else:
                do_highlight = True

        for section in parser_output:
            if do_highlight:
                section['contents'] = highlight(section['contents'], pyg_lexer, pyg_formatter)
            self.output_data._data.append(section)
        self.output_data.save()

Content © 2020 Dr. Ana Nelson | Site Design © Copyright 2011 Andre Gagnon | All Rights Reserved.