logo

docs >> guide >> creating-content

Creating Content

When we write template-based documents, we have a template, and information that fills in the blanks in that template. This chapter deals with how to make information available to your documents, and covers the standard information that Dexy makes available by default. A very simple example of creating a document from a template is string interpolation, a feature found in many programming languages. Here is an example in Python: 
data = {
    "foo" : 7,
    "bar" : "apple"
}

template = "foo is %(foo)s, bar is %(bar)s"

print template % data
Running this code will output the following 'document' with dynamic content replacing the placeholders: 
foo is 7, bar is apple
The two basic elements are: a document template with placeholders, and a source of data to fill in those placeholders. Dexy's plugin system provides a convenient way to make any kind of data available to your documents, including data from other documents. ### Template Plugins Dexy has a system of Template Plugins which allow us to easily customize what data we want available to our placeholders when we come to write documents. Here is the source of the base TemplatePlugin class: 
class TemplatePlugin(object):
    def __init__(self, filter_instance):
        self.filter_instance = filter_instance

    def run(self):
        return {}

Template plugins are very simple classes which implement a run method which returns a dictionary of key-value pairs. The values can be pretty much anything: primitives like numbers or strings, python functions or classes, or other data structures like dictionaries or arrays.

The base TemplateFilter class defines a run_plugins method which calls the run() method for each plugin listed in the PLUGINS constant, and aggregates all the results into 1 dict. Each TemplateFilter class can then use that dict to provide data to the documents which pass through that filter in a process or process_text method. The base TemplateFilter just uses string interpolation:

class TemplateFilter(DexyFilter):
    """
    Base class for templating system filters such as JinjaFilter. Templating
    systems are used to make generated artifacts available within documents.

    Plugins are used to prepare content.
    """
    ALIASES = ['template']
    FINAL = True
    PLUGINS = [
        ClippyHelperTemplatePlugin,
        DexyVersionTemplatePlugin,
        GlobalsTemplatePlugin,
        InputsTemplatePlugin,
        PrettyPrinterTemplatePlugin,
        PygmentsStylesheetTemplatePlugin,
        PythonBuiltinsTemplatePlugin,
        PythonDatetimeTemplatePlugin,
        RegularExpressionsTemplatePlugin,
        SubdirectoriesTemplatePlugin,
        VariablesTemplatePlugin
        ]

    def run_plugins(self):
        env = {}
        for plugin_class in self.plugins():
            plugin = plugin_class(self)
            new_env_vars = plugin.run()
            if any(v in env.keys() for v in new_env_vars):
                raise Exception("trying to add new keys %s, already have %s" % (u", ".join(new_env_vars.keys()), u", ".join(env.keys())))
            env.update(new_env_vars)
        return env

    def plugins(self):
        if self.artifact.args.has_key('plugins'):
            plugin_names = self.artifact.args['plugins']
            dict_items = sys.modules[self.__module__].__dict__
            return [dict_items[name] for name in plugin_names]
        else:
            return self.PLUGINS

    def process_text(self, input_text):
        """
        Overwrite this in your subclass, or, perhaps better, overwrite
        process() and write output directly to artifact file.
        """
        template_data = self.run_plugins()
        return input_text % template_data

The JinjaFilter passes the dict to the document template's stream method:

class JinjaFilter(JinjaTextFilter):
    """
    Runs the Jinja templating engine on your document to incorporate dynamic
    content.
    """
    ALIASES = ['jinja']
    TAGS = ['template']

    def process(self):
        env = self.setup_jinja_env()
        template_data = self.run_plugins()
        try:
            template = env.from_string(self.artifact.input_text())
            template.stream(template_data).dump(self.artifact.filepath(), encoding="utf-8")
        except (TemplateSyntaxError, UndefinedError, TypeError) as e:
            self.handle_jinja_exception(e)

This means that the different templating systems implemented by Dexy can all share the underlying data processing. So, it's easy to switch between templating systems, or to add support for a new one. And, you can create your own custom plugins, and also custom filters which can incorporate just the plugins you want.

Writing Custom Plugins and Templates

In this section, we look at writing some custom plugins and template filters, and using them in a document. Even if you aren't interested in writing custom plugins, this section will help you understand how plugins work, and how template filters make use of them.

Here is a simple plugin which just returns a static dict:

class SimpleTemplatePlugin(TemplatePlugin):
    def run(self):
        return { 'foo' : 42, 'bar' : 'tulip' }

We can create a template filter which uses this by subclassing the TemplateFilter class and just defining an alias and overriding the list of plugins, to include just our custom plugin.

class SimpleTemplateFilter(TemplateFilter):
    ALIASES = ['simple']
    PLUGINS = [ SimpleTemplatePlugin ]

Here are the files in this example:

.
./.dexy
./filters
./filters/__init__.py
./filters/__init__.pyc
./filters/simple_template_filter.py
./filters/simple_template_filter.pyc
./template.txt

The dexy config is:

{
    "template.txt|simple": {}
}

Our template document is:

Foo is %(foo)s. Bar is %(bar)s.

Here is the output generated:

Foo is 42. Bar is tulip.

Here is another filter, this time based on Jinja rather than string interpolation:

class SimpleTemplateFilter(JinjaFilter):
    ALIASES = ['simplejinja']
    PLUGINS = [ SimpleTemplatePlugin ]

Now our template looks like:

Foo is {{ foo }}. Bar is {{ bar }}.

Here are the files in this example:

.
./.dexy
./filters
./filters/__init__.py
./filters/__init__.pyc
./filters/simple_template_filter.py
./filters/simple_template_filter.pyc
./template.txt

And the output is:

Foo is 42. Bar is tulip.

Let's make another filter with some more interesting structures:

from dexy.filters.templating_filters import JinjaFilter
from dexy.filters.templating_plugins import TemplatePlugin

class CalculationResultPlugin(TemplatePlugin):
    def run(self):
        return { 'foo' : [(x, 2*x) for x in range(0, 10)] }

class FunctionsPlugin(TemplatePlugin):
    def double(self, x):
        return 2*x

    def triple(self, x):
        return 3*x

    def run(self):
        return { 'double' : self.double, 'triple' : self.triple }

class TwoPluginsTemplateFilter(JinjaFilter):
    ALIASES = ['twoplugins']
    PLUGINS = [ CalculationResultPlugin, FunctionsPlugin ]

{% for x, y in foo -%}
two times {{ x }} is {{ y }}
{% endfor -%}

double {{ 5 }} is {{ double(5) }}
triple {{ 5 }} is {{ triple(5) }}

Here are the files in this example:

.
./.dexy
./filters
./filters/__init__.py
./filters/__init__.pyc
./filters/simple_template_filter.py
./filters/simple_template_filter.pyc
./template.txt

And the output is:

two times 0 is 0
two times 1 is 2
two times 2 is 4
two times 3 is 6
two times 4 is 8
two times 5 is 10
two times 6 is 12
two times 7 is 14
two times 8 is 16
two times 9 is 18
double 5 is 10
triple 5 is 15

Plugins can be used to expose lots of information in documents. Commonly in Dexy, we will want to have access to the other documents we specified as inputs, and we can use plugins to customize how we display this information.

We create a plugin to load CSV data from an input file into a DictReader object:

from dexy.filters.templating_filters import JinjaFilter
from dexy.filters.templating_plugins import TemplatePlugin
import csv

class CsvPlugin(TemplatePlugin):
    def run(self):
        csv_files = {}
        for artifact_key, artifact in self.filter_instance.artifact.inputs().iteritems():
            f = open(artifact.filepath(), "r")
            csv_files[artifact.key] = csv.DictReader(f)
        return {'csv_rows' : csv_files}

class CsvInputTemplateFilter(JinjaFilter):
    ALIASES = ['csv']
    PLUGINS = [ CsvPlugin ]

Here is the template:

{% for artifact_key, rows in csv_rows|dictsort -%}
Rows in {{ artifact_key }}:
{% for row in rows -%}
{{ row }}
{% for k, v in row|dictsort -%}
{{ k }} : {{ row[k] }}
{% endfor -%}
{% endfor -%}
{% endfor -%}

Here are the files in this example:

.
./data1.csv
./data2.csv
./.dexy
./filters
./filters/__init__.py
./filters/__init__.pyc
./filters/simple_template_filter.py
./filters/simple_template_filter.pyc
./template.txt

Here is the config file:

{
    "template.txt|csv": {
        "inputs": [
            "*.csv"
        ]
    }
}

Here are the input files:

x,y,z
1,2,3
5,6,7


a,b,c
2,4,6
10,12,14

Here is the output:

Rows in data1.csv:
{'y': '2', 'x': '1', 'z': '3'}
x : 1
y : 2
z : 3
{'y': '6', 'x': '5', 'z': '7'}
x : 5
y : 6
z : 7
Rows in data2.csv:
{'a': '2', 'c': '6', 'b': '4'}
a : 2
b : 4
c : 6
{'a': '10', 'c': '14', 'b': '12'}
a : 10
b : 12
c : 14

The Built-In Plugins

The dict items defined in all of the following plugins are available in the default Dexy filters, so you can use them in your documents that are run through these filters.

If you want to disable some of these filters, for performance reasons or to avoid naming conflicts, then you can subclass a template filter and override the PLUGINS constant to just have the plugins you want, whether they are built-in or custom plugins.

The PrettyPrinterTemplatePlugin makes pformat available (the version of pretty print that prints to a string, rather than stdout):

class PrettyPrinterTemplatePlugin(TemplatePlugin):
    def run(self):
        return { 'pformat' : pprint.pformat}

Use regular expression matching and searching:

class RegularExpressionsTemplatePlugin(TemplatePlugin):
    def run(self):
        return { 're_match' : re.match, 're_search' : re.search}

Make most Python builtins available:

class PythonBuiltinsTemplatePlugin(TemplatePlugin):
    # Intended to be all builtins that make sense to run within a document.
    PYTHON_BUILTINS = [abs, all, any, bin, bool, bytearray, callable, chr,
        cmp, complex, dict, dir, divmod, enumerate, filter, float, format,
        hex, id, int, isinstance, issubclass, iter, len, list, long, map, hasattr,
        max, min, oct, ord, pow, range, reduce, repr, reversed, round,
        set, slice, sorted, str, sum, tuple, xrange, zip]

    def run(self):
        return dict((f.__name__, f) for f in self.PYTHON_BUILTINS)

Makes HTML and LaTeX Pygments stylesheets available:

class PygmentsStylesheetTemplatePlugin(TemplatePlugin):
    def highlight(self, text, lexer_name, fmt = 'html', noclasses = False):
        formatter_options = { "noclasses" : noclasses }
        lexer = pygments.lexers.get_lexer_by_name(lexer_name)
        formatter = pygments.formatters.get_formatter_by_name(fmt, **formatter_options)
        return pygments.highlight(text, lexer, formatter)

    def run(self):
        pygments_stylesheets = {}
        if self.filter_instance.artifact.args.has_key('pygments'):
            formatter_args = self.filter_instance.artifact.args['pygments']
        else:
            formatter_args = {}

        for style_name in get_all_styles():
            for formatter_class in [pygments.formatters.LatexFormatter, pygments.formatters.HtmlFormatter]:
                formatter_args['style'] = style_name
                pygments_formatter = formatter_class(**formatter_args)
                style_info = pygments_formatter.get_style_defs()

                for fn in pygments_formatter.filenames:
                    ext = fn.split(".")[1]
                    if ext == 'htm':
                        ext = 'css' # swap the more intuitive '.css' for the unlikely '.htm'
                    key = "%s.%s" % (style_name, ext)
                    pygments_stylesheets[key] = style_info
        return {'pygments' : pygments_stylesheets, 'highlight' : self.highlight }

Lists subdirectories of the directory a document is in:

class SubdirectoriesTemplatePlugin(TemplatePlugin):
    def run(self):
        # The directory containing the document to be processed.
        doc_dir = os.path.dirname(self.filter_instance.artifact.name)

        # Get a list of subdirectories under this document's directory.
        subdirectories = [d for d in sorted(os.listdir(os.path.join(os.curdir, doc_dir))) if os.path.isdir(os.path.join(os.curdir, doc_dir, d))]
        return {'subdirectories' : subdirectories}

Gives access to variables specified in a dexy config file:

class VariablesTemplatePlugin(TemplatePlugin):
    def run(self):
        variables = {}
        if self.filter_instance.artifact.args.has_key('variables'):
            variables.update(self.filter_instance.artifact.args['variables'])
        if self.filter_instance.artifact.args.has_key('$variables'):
            variables.update(self.filter_instance.artifact.args['$variables'])
        return variables

Gives access to globals specified in a dexy config file:

class GlobalsTemplatePlugin(TemplatePlugin):
    """
    Makes available the global variables specified on the dexy command line
    using the --globals option
    """
    def run(self):
        if self.filter_instance.artifact.controller_args.has_key('globals'):
            return self.filter_instance.artifact.controller_args['globals']
        else:
            return {}

Does preprocessing of inputs. Loads JSON into a dict etc. Populates the standard dexy objects:

class InputsTemplatePlugin(TemplatePlugin):
    def load_sort_json_data(self, a):
        try:
            unsorted_json = json.loads(a.output_text())
        except ValueError as e:
            print "unable to load JSON for", a.key
            print a.filename()
            print len(a.output_text())
            raise e

        def sort_dict(d):
            od = OrderedDict()
            for k in sorted(d.keys()):
                v = d[k]
                if isinstance(v, dict) or isinstance(v, OrderedDict):
                    od[k] = sort_dict(v)
                else:
                    od[k] = v
            return od

        if type(unsorted_json) == dict:
            return sort_dict(unsorted_json)
        else:
            return unsorted_json

    def run(self):
        d_hash = {}
        a_hash = {}

        name = self.filter_instance.artifact.name
        inputs = self.filter_instance.artifact.inputs()

        for key, a in inputs.iteritems():
            keys = a.relative_refs(name)

            # Do any special handling of data
            if a.ext == '.json':
                if len(a.output_text()) == 0:
                    # Hack for JSON data being written directly to a file, e.g. filenames filter
                    with open(a.filepath(), "rb") as f:
                        a.data_dict['1'] = f.read()
                data = self.load_sort_json_data(a)
            else:
                data = a.data_dict

            for k in keys:
                # Avoid adding duplicate keys
                if a_hash.has_key(k):
                    next

                a_hash[k] = a

                if hasattr(data, 'keys') and data.keys() == ['1']:
                    d_hash[k] = data['1']
                else:
                    d_hash[k] = data

        return {
            'a' : a_hash,
            's' : self.filter_instance.artifact,
            'd' : d_hash,
            'f' : self.filter_instance,
        }

Makes the 'clippy' helper available, so you can make it easy for readers to copy/paste text snippets:

class ClippyHelperTemplatePlugin(TemplatePlugin):
    PRE_AND_CLIPPY_STRING = """
<pre>
%s
</pre>%s
    """
    CLIPPY_HELPER_STRING = """
<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
                width="110"
                height="14"
                id="clippy" >
<param name="movie" value="/clippy.swf"/>
<param name="allowScriptAccess" value="always" />
<param name="quality" value="high" />
<param name="scale" value="noscale" />
<param NAME="FlashVars" value="text=%s">
<param name="bgcolor" value="#ffffff">
<embed src="/clippy.swf"
       width="110"
       height="14"
       name="clippy"
       quality="high"
       allowScriptAccess="always"
       type="application/x-shockwave-flash"
       pluginspage="http://www.macromedia.com/go/getflashplayer"
       FlashVars="text=%s"
       bgcolor="#ffffff"
/>
</object>"""

    def run(self):
        return {
            'pre_and_clippy' : self.pre_and_clippy,
            'clippy_helper' : self.clippy_helper
        }

    def pre_and_clippy(self, text):
        return self.PRE_AND_CLIPPY_STRING % (text, self.clippy_helper(text))

    def clippy_helper(self, text):
        if not text or len(text) == 0:
            raise Exception("You passed blank text to clippy helper!")
        quoted_text = urllib.quote(text)
        return self.CLIPPY_HELPER_STRING % (quoted_text, quoted_text)
blog comments powered by Disqus

This website was generated by Dexy. | This Page's Source | This Page's Log (large HTML page) | Back to Top

Quick Note

About Dexy

Dexy is a free-form literate documentation tool for writing any kind of technical document incorporating code. Dexy helps you write correct documents, and to easily maintain them over time as your code changes.

Latest News

From Twitter

  •  
Follow Dexy On Twitter

Thanks To

Dexy Sponsors

Content © 2011 Dr. Ana Nelson | Site Design © Copyright 2011 Andre Gagnon | All Rights Reserved.