logo

Getting Started

This section is designed to gently introduce you to Dexy by working through several progressive examples. These examples are written in Python because everyone using Dexy will have Python installed on their system, and Python is an easy, accessible language to understand. However, Dexy supports many other programming languages and it's easy to add support for new languages, so please don't feel left out if Python is not your preferred language.

As a reminder, if you aren't already familiar with using the command line, then you should work through a course like Zed Shaw's Command Line Crash Course before you start with these examples.

The point of all these examples is for you to work through them, not just read them. While the contents of each example can be found in a git repository, you will get far more benefit from typing in the contents manually. The content is short, and typing it in will mean you will be more aware of the details of the syntax, and you will make typos and mistakes and learn how dexy responds and how to debug and fix them. After a while the examples will stop telling you to make new files and type them in, but you are still expected and encouraged to do so. Also, please experiment once you have understood the basics, and you are welcome to use these examples as the starting point for your own projects.

Simple Dexy Example

We are going to create a very simple project to learn how to run dexy.

Create a new, empty directory to work in. Make a text file named hello.txt and type the following into it:

Hi, welcome to Dexy

1 + 1 = {{ 1 + 1 }}

The curly brackets are jinja tags, and they are how we are going to get dexy to incorporate code and other fun stuff into our documents later on. Whatever goes in those brackets gets evaluated, including simple arithmetic like 1+1.

Next we need to give dexy some instructions about what files we want to be processed, and what we want done to those files. Create another text file named dexy.txt (the name is important) and type the following into it:

hello.txt|jinja

Here we are telling dexy to take the file named hello.txt and run it through the jinja filter.

Your directory should now look like this:

$ ls
dexy.txt  hello.txt

Now we are ready to run dexy! The first time you run dexy in a directory, you will need to call dexy setup first. Then to run dexy you simply type dexy:

$ dexy setup
$ dexy
dexy run finished in 0.225
dexy reports finished in 0.139

Dexy puts some of the finished documents into the output/ directory (the ones it thinks you are most interested in) and all the finished documents into the output-long/ directory (with uglier, but unique, filenames). To see the output of our hello.txt template, you can run:

$ cat output/hello.txt
Hi, welcome to Dexy

1 + 1 = 2

We see from this output that jinja has taken the 1+1 which we typed within the tag and evaluated it, giving 2. So, this tells us that dexy has successfully called the jinja filter we specified in the dexy.txt file.

Every time you change a file and want dexy to re-generate the output, you need to call the dexy command again. As this tutorial goes on, it will stop reminding you to run the dexy command, but you'll need to do so each time.

The Run Report

Dexy has also generated a report which describes what has happened. You should have a HTML file at .dexy/reports/run/index.html in your project directory.

You can open this HTML file in a web browser to view it. It's in a hidden directory so you have to set the "show hidden files" setting to see it in your file browser (search online for instructions on how to do this for your specific operating system), or use a command line tool like open or sensible-browser to open the file in your browser from the command line.

The run report lets you drill down into the steps of dexy's processing. It's especially useful when the output of a document isn't written to the output/ directory as we'll see in the next section. As you work on a project, keep the report open in a browser and refresh the page each time you run dexy.

Getting Dexy to Run Code

Next, we will get dexy to evaluate a simple python script. In the same directory where you've been working, create a new file named hello.py and put the following in it:

x = 6
y = 7
print x * y

Your project directory should look like this now:

$ ls
dexy.txt  hello.py  hello.txt

Run this script from the command line to make sure that it works, you should see the script print out the results of the calculation:

$ python hello.py
42

Now modify your dexy.txt file so that it looks like this:

hello.py|py
hello.txt|jinja

This means we want the file named hello.py to be run through the py filter, and we want the file named hello.txt to be run through the jinja filter. The order of these lines is important because in the next section we want hello.py to be run first, so that it is available to hello.txt|jinja.

Now you can call dexy again:

$ dexy
dexy run finished in 0.264
dexy reports finished in 0.146

This time we can't look in the output/ directory to see what dexy did, because by default the output of running code through the py filter isn't included in that directory. However, we can see the results in the run report. You should use the run report to verify that dexy has run the code and produced the expected result.

Showing Code and Output

Now we're finally ready to start using dexy to document code! In this example we will incorporate the python script we wrote into our hello.txt document, and we will also show the output it produces. Modify the dexy.txt file so it looks like this:

hello.py|py
hello.py
hello.txt|jinja

And modify the hello.txt file so that it looks like this:

Here is some code:

{{ d['hello.py'] }}

Here is the output when run:

{{ d['hello.py|py'] }}

Now try running dexy, and then view the contents of the output hello.txt file:

$ cat output/hello.txt
Here is some code:

x = 6
y = 7
print x * y


Here is the output when run:

42

If this doesn't work, go back and make sure you have typed everything in exactly as shown (exactly means every single character is identical, even spaces and blank lines).

Next, experiment with changing one of the variable values in hello.py, for example set x to 8, and then run dexy again. The output/hello.txt file will be updated with the new value in both the script and the output.

Here are some more things you can try:

  • Change the text in your hello.txt file and run dexy again.
  • Remove just one of the closing curly brackets } from the hello.txt file and try to run dexy again. You should get an error message because the jinja processor can't parse the file. Fix the file and make sure dexy runs with no more errors. Experiment with other ways to break things, noting that some produce error messages and others don't (like removing an opening curly bracket {).

Processing Multiple Scripts

Next we're going to add a second python script and make a change to the way we write config files.

Change your dexy.txt file so that it looks like this:

.py
.py|py
hello.txt|jinja

We have replaced the file name hello.py with a wildcard expression which will match any file ending in .py. After making this change, run dexy and make sure everything still works.

Next create a new file named loop.py and put the following into it:

for i in range(0, 5):
    print i

And then modify your hello.txt file to look like this:

Here is some Python code:

{{ d['hello.py'] }}

Here is the output from running this script:

{{ d['hello.py|py'] }}

Here is some Python code which iterates through some numbers:

{{ d['loop.py'] }}

Here is the output from running this script:

{{ d['loop.py|py'] }}

Then after you have run dexy, the contents of output/hello.txt should look like this:

$ cat output/hello.txt
Here is some Python code:

x = 6
y = 7
print x * y


Here is the output from running this script:

42


Here is some Python code which iterates through some numbers:

for i in range(0, 5):
    print i


Here is the output from running this script:

0
1
2
3
4

YAML Configuration Files

Up to now we have been creating files named dexy.txt and listing a few documents in each file to tell dexy what to do. This was a simple way to get started, but it has limitations. Now we want to start using a YAML-based configuration file format. YAML is a relatively sane and human-friendly format.

Delete your dexy.txt file and create a new file dexy.yaml with contents:

hello.txt|jinja:
    - .py|py
    - .py

Run the dexy command again, you should get the same result as before.

The way you use the YAML syntax is to list the dependencies for a document underneath it in an indented list. As a shortcut, you can (and should) leave off the initial * for a wildcard expression. (If you ever do need to start an expression with an asterisk, then it needs to be wrapped in "double quotes" or escaped with a \ (forward slash).) In general in YAML you do not need to put string expressions in quotes (which makes it very convenient to work with). You can also include comments in your YAML by starting a comment line with #.

HTML Documents

Now we will start creating HTML documents instead of just plain text, and we'll also learn about some of the other filters we can use to run python code.

Create a new working directory. Let's start by writing a short Python script called script.py:

for i in range(0, 5):
    print i

And also create a simple HTML file named doc.html which includes the source of the python file:

<html>
    <head>
    </head>
    <body>

<p>Here is a Python script:</p>
<pre>
{{ d['script.py'] }}
</pre>

    </body>
</html>

Here is the dexy.yaml configuration:

.html|jinja:
    - .py

Your working directory should look like this:

$ ls
dexy.yaml  doc.html  script.py

Because this is a new project, we need to call dexy setup once before we call dexy:

$ dexy setup
$ dexy
dexy run finished in 0.382
dexy reports finished in 0.270

The generated HTML should be:

$ cat output/doc.html
<html>
    <head>
    </head>
    <body>

<p>Here is a Python script:</p>
<pre>
for i in range(0, 5):
    print i

</pre>

    </body>
</html>

If you open the file in a browser, it will look like:

Syntax Highlighting

Now that we're using HTML, let's make this output a little more colorful by applying syntax highlighting to our source code. Here's how you include this in your HTML:

<html>
    <head>

        <style type="text/css">
            {{ pygments['pastie.css'] }}
        </style>

    </head>
    <body>

        <p>Here is a Python script:</p>
        {{ d['script.py|pyg'] }}

    </body>
</html>

In the header of the file, we are inserting style definitions into a text/css style block. The 'pygments' object we use is a dict which contains CSS (and also LaTeX) stylesheets in various styles. Just pass the name of the style with the appropriate file extension to include it in your HTML header. Also make sure to add the |pyg after script.py in the body of the html document.

Next, change the dexy.yaml file to look like:

.html|jinja:
    - .py|pyg

After you run this example, open the file in a web browser, you should see the source code colorized.

Next we want to run the python code. Add a line to the dexy.yaml file:

.html|jinja:
    - .py|pyg
    - .py|py

And update the html file:

<html>
    <head>
        <style type="text/css">
            {{ pygments['pastie.css'] }}
        </style>
    </head>
    <body>

        <p>Here is a Python script:</p>
        {{ d['script.py|pyg'] }}

<p>Here is the output:</p>
<pre>
{{ d['script.py|py'] }}
</pre>

    </body>
</html>

Passing Custom Options

Now let's pass a custom option to the pyg filter:

.html|jinja:
    - .py|pyg:
        - pyg: { linenos : inline }
    - .py|py

To pass custom options to a filter, add an indented line beneath the document and start with the filter alias, followed by a colon, then the dict of options. The filter documentation should tell you what available options are.

There is no need to make any change to the HTML file. After running dexy you should see line numbers appear in the generated doc.html.

Next, look at the documentation for the pygments HtmlFormatter and try out some of the other options.

Applying a HTML Template

In the last few examples we have been writing complete HTML documents by hand, but typing <head> tags all the time gets old fast. So, now let's use another dexy filter to help us.

We will use the easyhtml filter in dexy to apply a basic stylesheet including pygments CSS to our document. The dexy.yaml file should look like this:

.html|jinja|easyhtml:
    - .py|pyg:
        - pyg: { linenos : inline }
    - .py|py

Remove everything from the doc.html file except the contents of the <body> tags, it should look like this now:

<p>Here is a Python script:</p>
{{ d['script.py|pyg'] }}

<p>Here is the output:</p>
<pre>
{{ d['script.py|py'] }}
</pre>

Now we are applying multiple filters to the doc.html file. First, we run the jinja filter. Second, we run the easyhtml filter which adds a header and footer to our document, making it a complete HTML document.

Python Console Output

The pycon filter used in this section is not available for Windows. If you are using Windows to run dexy then the example described in this section will not work.

Now, let's change this example so that instead of showing the code and, separately, showing the output, we just show a console transcript. The dexy.yaml file should look like:

.html|jinja|easyhtml:
    - .py|pycon|pyg:
        - pyg: { linenos : inline }

The pycon dexy filter runs python code in the python REPL, so you see the prompts, input and output from running each line of code. We can pass this REPL transcript to pygments which knows how to syntax highlight console output.

Update the html file as follows:

Here is a Python script:

{{ d['script.py|pycon|pyg'] }}

Splitting Code into Sections

Up until now we have been running whole python scripts. However, we don't want to always have to include whole scripts in our documents. Dexy is designed to allow you to split your code into sections and then preserve these sections in subsequent filters.

The idio filter will interpret special comments in your source code and split your script into sections accordingly. Create a new working directory and create a file named script.py which should look like this:

### @export "assign-variables"
x = 6
y = 7

### @export "multiply"
print x*y

These comments follow a special format of three comment characters, the python comment character being #, followed by the @export command, and then the name of the section in quotes. We have defined two sections, the first named assign-variables and the second named multiply.

Here is the dexy.yaml file which tells dexy to run all files with .py extension through the idio filter:

doc.html|jinja|easyhtml:
    - .py|idio
    - .py|py

Then in our document, we refer to the sections as follows:

<p>First, we assign variables:</p>

{{ d['script.py|idio']['assign-variables'] }}

<p>Then, we multiply:</p>

{{ d['script.py|idio']['multiply'] }}

<p>The output is:</p>

<pre>
{{ d['script.py|py'] }}
</pre>

By default, the idio filter will apply HTML syntax highlighting to the sections. So, you can include the output directly in HTML documents. To prevent idio from adding HTML formatting, add the t filter after it. The t filter will only accept input files that end with the .txt extension, so this forces idio to generate plain text output:

doc.html|jinja|easyhtml:
    - .py|idio|t
    - .py|py

Now we have to wrap the sections in <pre> tags:

<p>First, we assign variables:</p>

<pre>
{{ d['script.py|idio|t']['assign-variables'] }}
</pre>

<p>Then, we multiply:</p>

<pre>
{{ d['script.py|idio|t']['multiply'] }}
</pre>

<p>The output is:</p>

<pre>
{{ d['script.py|py'] }}
</pre>

Running Sectioned Code

The pycon filter used in this section is not available for Windows. If you are using Windows to run dexy then the example described in this section will not work.

Splitting code into sections is really useful when we can pass this code through an interpreter, such as the pycon filter, and keep the sections. Here is the dexy.yaml:

doc.html|jinja|easyhtml:
    - .py|idio|pycon|pyg

We pass our python script through 3 filters. First, the idio filter will split the code into sections. Second, the pycon filter will run the code through the python interpreter (the pycon filter accepts files ending in .py or .txt extensions, so this forces the idio filter to output plain text). Finally, the pyg filter will apply syntax highlighting to the output from the python interpreter.

Our doc.html looks like:

<p>First, we assign variables:</p>

{{ d['script.py|idio|pycon|pyg']['assign-variables'] }}

<p>Then, we multiply:</p>

{{ d['script.py|idio|pycon|pyg']['multiply'] }}

Using Markup Languages

The next thing we want to be able to do is to generate HTML without having to type all of the HTML tags ourselves. There are several lightweight markup languages commonly in use, such as Markdown, reStructuredText, Wiki markup (various flavors), and AsciiDoc.

The examples that follow will use reStructuredText since dexy already comes with the software needed to generate various output formats from rst files.

Here is the dexy.yaml:

doc.rst|jinja|rst:
    - .py|idio|t
    - .py|py

The rst filter takes rst and converts it into one of reStructuredText's output formats. By default it will output self-contained HTML documents, which is what we want.

Create a file named doc.rst with these contents:

Code
----

First, we assign variables::

    {{ d['script.py|idio|t']['assign-variables'] | indent(4) }}

Then, we multiply::

    {{ d['script.py|idio|t']['multiply'] | indent(4) }}

Output
------

The output is::

    {{ d['script.py|py'] | indent(4) }}

If you aren't familiar with reStructuredText, you can work through the quickstart and then refer to the quickref. For more advanced usage, the markup specification describes the language in detail and the directives reference describes various directives you can include in reStructuredText documents such as .. image:: and ..table::.

In this example, we create two sections by underlining the section names with hyphens. To indicate that our code samples and the generated output is preformatted, we end the preceding paragraphs with ::. We have indented the jinja tags by 4 spaces, but after jinja inserts its contents, only the first line will be properly indented. Fortunately, jinja comes with an indent filter, and we indicate that we want text indented by 4 spaces (this is the default, so it could be omitted). By default, jinja's indent filter assumes you have indented the first line manually, as we have here, so it won't end up being double-indented.

Here is what the resulting doc.html file looks like:

Syntax Highlighting: Using RST Directives

In the previous section, we simply indicated that our code samples were preformatted, so they appeared in a fixed-width font. Now we want to add syntax highlighting. There are a few ways to apply syntax highlighting and they have different implications, so there will be a few sections about this topic.

In this example, we will use reStructuredText's built-in syntax highlighting. The .. code:: directive tells reStructuredText to apply syntax highlighting to the subsequent indented block of text.

We only need to modify the doc.rst document. Now we end the preceding paragraphs with just a single : instead of two, and we add the .. code:: directive, specifying that the language to be used is python. We do not need to change the contents of our jinja tags:

Code
----

First, we assign variables:

.. code:: python

    {{ d['script.py|idio|t']['assign-variables'] | indent(4) }}

Then, we multiply:

.. code:: python

    {{ d['script.py|idio|t']['multiply'] | indent(4) }}

Output
------

The output is::

    {{ d['script.py|py'] | indent(4) }}

Here is what the resulting doc.html file looks like:

Customizing Builtin Highlighting

reStructuredText allows you to customize the behavior of a directive (like .. code::) by specifying directive options. Directive options take the form of field lists.

Here is the doc.rst file with the number-lines directive option specified for the first python code block:

Code
----

First, we assign variables:

.. code:: python
   :number-lines: 1

    {{ d['script.py|idio|t']['assign-variables'] | indent(4) }}

Then, we multiply:

.. code:: python

    {{ d['script.py|idio|t']['multiply'] | indent(4) }}

Output
------

The output is::

    {{ d['script.py|py'] | indent(4) }}

We are using reStructuredText's default HTML template which includes a stylesheet for the syntax highlighting. Unfortunately there is no configuration option which allows you to quickly specify a different pygments style, you need to specify a completely different template. This can be specified by passing configuration options to reStructuredText, however we will see shortly how to use a custom HTML template using dexy which will be easier.

Here is how to pass configuration options to reStructuredText:

doc.rst|jinja|rst:
    - rst: { initial-header-level: 3, id_prefix: foo }
    - .py|idio|t
    - .py|py

You can use hyphenated or underscore syntax, so initial-header-level and initial_header_level will both work. You can see all the available configuration options by running rst2html.py --help or viewing the configuration documentation.

Here is what the resulting doc.html file looks like, with line numbers enabled on the first code example and with the custom configuration options:

Typing Less with Macros

Let's look again at our reStructuredText document from the previous section:

Code
----

First, we assign variables:

.. code:: python
   :number-lines: 1

    {{ d['script.py|idio|t']['assign-variables'] | indent(4) }}

Then, we multiply:

.. code:: python

    {{ d['script.py|idio|t']['multiply'] | indent(4) }}

Output
------

The output is::

    {{ d['script.py|py'] | indent(4) }}

While the reStructuredText code directive syntax is pretty concise, it's still a lot of extra typing, especially if we have a long document. Dexy is all about automation, so let's see if we can shorten the amount of text needed to include a block of source code.

We are using the jinja templating system to incorporate content into our reStructuredText documents. Jinja supports defining custom macros, so we will use a macro to help simplify creating blocks of code.

Create a new file named rst.jinja with contents:

{% macro rst_code(document_key, section_name, language="python", number_lines=False) %}
.. code:: {{ language }} {% if number_lines %}
   :number-lines: 1 {% endif %}

    {{ d[document_key][section_name] | indent(4) }}

{% endmacro %}

reStructuredText is very fussy about whitespace, so when writing a macro to generate a directive you may have to do some fiddling to get the whitespace right. A good way to do this is to just run the jinja filter and not the rst filter until you have generated the correct syntax. See the section in the jinja template documentation about whitespace control for more information.

To use the macro, we need to import it into our document template before the first usage:

Code
----

{% from 'rst.jinja' import rst_code with context -%}

First, we assign variables:

{{ rst_code('script.py|idio|t', 'assign-variables', number_lines=True) }}

Then, we multiply:

{{ rst_code('script.py|idio|t', 'multiply') }}

Output
------

The output is::

    {{ d['script.py|py'] | indent(4) }}

We no longer need to do any indenting in our document since this is handled in the macro. We just call the name of the macro and pass in the document key and section name, and optional keyword arguments if we want to change the language or whether lines are numbered.

The jinja filter in dexy automatically makes any macro definition files available to your documents, you just need to use the correct relative path from your document to the macro file. In this case rst.jinja is in the same directory as doc.rst.

Here is what the resulting doc.html file looks like:

Generating Multiple Output Formats

We have seen how we can generate HTML from reStructuredText source, but reStructuredText supports several other output formats too, and we can use dexy to generate all of these simultaneously.

Here is a dexy.yaml file which specifies that we want doc.rst converted to HTML, to PDF (via LaTeX), and to ODT (word processor) format:

code:
    - .py|idio|t
    - .py|py

# HTML output
doc.rst|jinja|rst:
    - code

# PDF output - requires a latex compiler on your system
doc.rst|jinja|rst|l|latex:
    - code

# ODT output - OpenOffice's word processor
# we end this with an 'alias' filter of '-odt' to avoid name collision
doc.rst|jinja|rst|-odt:
    - rst: { 'ext' : '.odt' }
    - code

Here are links to the resulting PDF and ODT files. Here is what the resulting doc.html file looks like:

Syntax Highlighting: Using Dexy

In the last few sections we have used reStructuredText's directives for applying syntax highlighting to code blocks. We've also been using reStructuredText's default HTML template. In this section we'll use a different approach where we'll do the syntax highlighting in dexy, and we'll use dexy to apply a template to the HTML. Either approach works, they each have pros and cons. It's common in dexy for there to be several ways to accomplish a given goal.

Here is the dexy.yaml we will use:

doc.rst|jinja|rstbody|easyhtml:
    - .py|idio
    - .py|py

We are now using a different filter, rstbody instead of rst. The rstbody filter does not apply a template, it just returns the body text converted to the desired output format, by default HTML. Then we use the easyhtml filter we've already seen to apply a template.

We will use the idio filter to split the python code into sections and syntax highlight them. Now we need to tell reStructuredText that we will be including chunks of HTML-formatted code which should be left alone. To do this we will use the raw directive:

Code
----

First, we assign variables:

.. raw:: html

    {{ d['script.py|idio']['assign-variables'] | indent(4) }}

Then, we multiply:

.. raw:: html

    {{ d['script.py|idio']['multiply'] | indent(4) }}

Output
------

The output is::

    {{ d['script.py|py'] | indent(4) }}

Here is what the resulting doc.html file looks like:

Macros Again

Once again, we can use a macro to simplify this. This time we'll call the macro codes which you can think of as standing for 'code - sectioned'.

Here is the rst.jinja:

{% macro codes(document_key, section_name) %}
.. raw:: html

    {{ d[document_key][section_name] | indent(4) }}
{% endmacro %}

And here is the doc.rst:

Code
----

{% from 'rst.jinja' import codes with context -%}

First, we assign variables:

{{ codes('script.py|idio', 'assign-variables') }}

Then, we multiply:

{{ codes('script.py|idio', 'multiply') }}

Output
------

The output is::

    {{ d['script.py|py'] | indent(4) }}

Here is what the resulting doc.html file looks like:

Generalizing Beyond HTML

As you may have noticed, by including raw HTML markup we have made it impossible to convert the reStructuredText to any format besides HTML. You can't generate LaTeX with a bunch of <span> tags in the middle of it and expect it to compile. Fortunately, we have a very nice way around this, which is to use macros not just to shorten what we have to type, but to be smart about what format to insert.

These macros are actually built into dexy, so you don't need to have an rst.jinja file of your own (although you can write one if you want to customize the behavior of the macros).

Here is the dexy.yaml we start with, we want to tell dexy to generate both HTML and LaTeX formatted syntax highlighting:

code:
    - .py|idio|h
    - .py|idio|l
    - .py|py

doc.rst|jinja|rstbody|easyhtml:
    - code

doc.rst|jinja|rstbody|easylatex|latex:
    - code

Here is our doc.rst file. We start by importing the codes macro from dexy.jinja which is a macro file which ships with dexy:

Code
----

{% from 'dexy.jinja' import codes with context -%}

First, we assign variables:

{{ codes('script.py|idio', 'assign-variables') }}

Then, we multiply:

{{ codes('script.py|idio', 'multiply') }}

Output
------

The output is::

    {{ d['script.py|py'] | indent(4) }}

Notice that we pass script.py|idio as the first argument to codes. The macro will look at the final output format of the document and insert contents from either script.py|idio|h or script.py|idio|l as required.

Here is the resulting PDF. Here is what the resulting doc.html file looks like:

Content © 2013 Dr. Ana Nelson | Site Design © Copyright 2011 Andre Gagnon | All Rights Reserved.