We will cover how to write .dexy config files, and what is happening behind the scenes when you define documents and dependencies.
Dexy works by taking file matching patterns you specify, finding matching files, and running these files through 0 or more filters, possibly also taking input from other documents if you have specified dependencies.
At the beginning of a Dexy run, the Controller makes a list of all the documents it needs to run, and the dependencies between these documents. It runs a topological sort to figure out what order it needs to run the documents in.
Let's look at some examples to see how this works.
To start with, we'll have a Python file, hello.py, an R file, hello.R, and a Text file, hello.txt. We won't run them through any filters at first. If you specify a file without any filters, Dexy just outputs the original file.
Here is our configuration file:
Dexy's controller will process this configuration file, looking for files that match the specified names and creating Document objects for each one. Normally, this happens as part of the controller's run() method which is called when you run the dexy command. However, to illustrate what's happening, here is a script which creates a Controller class and processes the configuration file:
controller = Controller(args)
controller.log = log
controller.load_config()
controller.process_config()
controller.virtual_docs = []
Running this code produces this output:
batch id is 1 sorting 3 documents into run order, there are 0 total dependencies
We have three documents, and they don't have any dependencies. As Dexy starts processing the config file, it creates Document objects for each document it will run, and it puts them into an OrderedDict called 'members', referenced by their document keys. We can loop over this members dict and see what's there:
for document_key, document in controller.members.iteritems():
s.write("%s : %s\n" % (document_key, document))
hello.txt : <dexy.document.Document object at 0x1ed0f10> hello.py : <dexy.document.Document object at 0x24be650> hello.R : <dexy.document.Document object at 0x24be790>
We can peek at the dependencies list to confirm that it's empty:
[]
Here is what gets written to the Dexy log:
2012-03-14 03:32:01,715 - ex1.log - DEBUG - project root excluded directories ignore, output-long, output, ../../../../artifacts, logs
2012-03-14 03:32:01,715 - ex1.log - DEBUG - directories excluded at all levels .bzr, .hg, .git, .svn
2012-03-14 03:32:01,715 - ex1.log - INFO - loading config file .dexy
2012-03-14 03:32:01,716 - ex1.log - INFO - loading config file ./.dexy
2012-03-14 03:32:01,716 - ex1.log - INFO - loading config file .dexy
2012-03-14 03:32:01,716 - ex1.log - INFO - loading config file ./.dexy
2012-03-14 03:32:01,716 - ex1.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.txt' args : '{'$variables': {}}'
2012-03-14 03:32:01,717 - ex1.log - DEBUG - creating doc hello.txt for glob hello.txt
2012-03-14 03:32:01,717 - ex1.log - DEBUG - no existing key hello.txt
2012-03-14 03:32:01,717 - ex1.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,717 - ex1.log - DEBUG - updating with: '{'$variables': {}}'
2012-03-14 03:32:01,717 - ex1.log - DEBUG - Doc args after update: '{'$variables': {}}'
2012-03-14 03:32:01,717 - ex1.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.py' args : '{'$variables': {}}'
2012-03-14 03:32:01,718 - ex1.log - DEBUG - creating doc hello.py for glob hello.py
2012-03-14 03:32:01,718 - ex1.log - DEBUG - no existing key hello.py
2012-03-14 03:32:01,718 - ex1.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,718 - ex1.log - DEBUG - updating with: '{'$variables': {}}'
2012-03-14 03:32:01,718 - ex1.log - DEBUG - Doc args after update: '{'$variables': {}}'
2012-03-14 03:32:01,718 - ex1.log - DEBUG - Parsing doc path: '.' input_dir: '$globals' args : '{'$variables': {}}'
2012-03-14 03:32:01,718 - ex1.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.R' args : '{'$variables': {}}'
2012-03-14 03:32:01,719 - ex1.log - DEBUG - creating doc hello.R for glob hello.R
2012-03-14 03:32:01,719 - ex1.log - DEBUG - no existing key hello.R
2012-03-14 03:32:01,719 - ex1.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,719 - ex1.log - DEBUG - updating with: '{'$variables': {}}'
2012-03-14 03:32:01,719 - ex1.log - DEBUG - Doc args after update: '{'$variables': {}}'
2012-03-14 03:32:01,719 - ex1.log - DEBUG - Parsing doc path: '.' input_dir: '$variables' args : '{'$variables': {}}'
2012-03-14 03:32:01,719 - ex1.log - DEBUG - Finalizing dependencies between documents...
2012-03-14 03:32:01,719 - ex1.log - DEBUG - finalizing dependencies for hello.txt
2012-03-14 03:32:01,719 - ex1.log - DEBUG - finalizing dependencies for hello.py
2012-03-14 03:32:01,719 - ex1.log - DEBUG - finalizing dependencies for hello.R
2012-03-14 03:32:01,719 - ex1.log - DEBUG - Beginning topological sort...
2012-03-14 03:32:01,719 - ex1.log - DEBUG - Topological sort completed successfully.
In the next section, we'll actually add a dependency to make things more interesting.
We just make 1 change to our configuration file:
We add an 'inputs' attribute to the "hello.txt" document.
The format of a .dexy configuration file is a JSON Object. A JSON Object is like a dictionary or hashmap, you have keys and values. In the case of a .dexy config, the keys are document keys, and the values are JSON Objects which can be empty (e.g. '{}'), or which can have additional arguments relating to that document. Some attributes are processed by Dexy itself, other attributes are passed on to Dexy filters.
The "inputs" attribute is one of the ways we tell Dexy that a document depends on other documents. In this case, we are telling Dexy that the "hello.txt" document depends on the "hello.py" and "hello.R" documents.
Now when we run process_config(), we should see output like this:
batch id is 1 sorting 3 documents into run order, there are 2 total dependencies ratio of dependencies to documents is 0.7
The members dict is:
hello.py : <dexy.document.Document object at 0x24bef10> hello.R : <dexy.document.Document object at 0x24bee50> hello.txt : <dexy.document.Document object at 0x24bef50>
This OrderedDict keeps track of members in the order they are added to the dict. The ordering is arbitrary, what's important is that each document has a consistent index within the dict.
Now, we can look at the dependencies list. This is a list of tuples, where each tuple (a, b) represents a dependency of document b on input document a.
Here is the dependency list for this example:
[(0, 2), (1, 2)]
So document 2 depends on document 0, and document 2 depends on document 1.
Dexy applies a topological sort to this dependency list to determine an order to run the documents so that, if b depends on a, a is always run before b.
Here is the ordering for this example:
[0, 1, 2]
This ordering is used to create a new list of the documents in the order in which they should be run by Dexy to satisfy all dependencies (if there are circular dependencies, Dexy raises an exception). Each document has a list of its inputs, which it will use later to access the information contained in the input documents.
for doc in controller.docs:
s.write(doc.key())
s.write(" inputs: %s\n" % repr([d.key() for d in doc.inputs]))
hello.py inputs: [] hello.R inputs: [] hello.txt inputs: [u'hello.py', u'hello.R']
We can see that by the time we reach hello.txt, we have already processed the two documents it depends upon.
Here is what gets written to the Dexy log:
2012-03-14 03:32:01,721 - ex2.log - DEBUG - project root excluded directories ignore, output-long, output, ../../../../artifacts, logs
2012-03-14 03:32:01,722 - ex2.log - DEBUG - directories excluded at all levels .bzr, .hg, .git, .svn
2012-03-14 03:32:01,722 - ex2.log - INFO - loading config file .dexy
2012-03-14 03:32:01,722 - ex2.log - INFO - loading config file ./.dexy
2012-03-14 03:32:01,722 - ex2.log - INFO - loading config file .dexy
2012-03-14 03:32:01,722 - ex2.log - INFO - loading config file ./.dexy
2012-03-14 03:32:01,722 - ex2.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.txt' args : '{u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}'
2012-03-14 03:32:01,722 - ex2.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.py' args : '{}'
2012-03-14 03:32:01,723 - ex2.log - DEBUG - creating doc hello.py for glob hello.py
2012-03-14 03:32:01,723 - ex2.log - DEBUG - no existing key hello.py
2012-03-14 03:32:01,723 - ex2.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,723 - ex2.log - DEBUG - updating with: '{}'
2012-03-14 03:32:01,723 - ex2.log - DEBUG - Doc args after update: '{}'
2012-03-14 03:32:01,723 - ex2.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.R' args : '{}'
2012-03-14 03:32:01,723 - ex2.log - DEBUG - creating doc hello.R for glob hello.R
2012-03-14 03:32:01,723 - ex2.log - DEBUG - no existing key hello.R
2012-03-14 03:32:01,723 - ex2.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,723 - ex2.log - DEBUG - updating with: '{}'
2012-03-14 03:32:01,723 - ex2.log - DEBUG - Doc args after update: '{}'
2012-03-14 03:32:01,724 - ex2.log - DEBUG - creating doc hello.txt for glob hello.txt
2012-03-14 03:32:01,724 - ex2.log - DEBUG - no existing key hello.txt
2012-03-14 03:32:01,724 - ex2.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,724 - ex2.log - DEBUG - updating with: '{u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}'
2012-03-14 03:32:01,724 - ex2.log - DEBUG - Doc args after update: '{u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}'
2012-03-14 03:32:01,724 - ex2.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.py' args : '{'$variables': {}}'
2012-03-14 03:32:01,724 - ex2.log - DEBUG - creating doc hello.py for glob hello.py
2012-03-14 03:32:01,724 - ex2.log - DEBUG - found existing key hello.py
2012-03-14 03:32:01,724 - ex2.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,724 - ex2.log - DEBUG - updating with: '{'$variables': {}}'
2012-03-14 03:32:01,724 - ex2.log - DEBUG - Doc args after update: '{'$variables': {}}'
2012-03-14 03:32:01,724 - ex2.log - DEBUG - Parsing doc path: '.' input_dir: '$globals' args : '{'$variables': {}}'
2012-03-14 03:32:01,724 - ex2.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.R' args : '{'$variables': {}}'
2012-03-14 03:32:01,725 - ex2.log - DEBUG - creating doc hello.R for glob hello.R
2012-03-14 03:32:01,725 - ex2.log - DEBUG - found existing key hello.R
2012-03-14 03:32:01,725 - ex2.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,725 - ex2.log - DEBUG - updating with: '{'$variables': {}}'
2012-03-14 03:32:01,725 - ex2.log - DEBUG - Doc args after update: '{'$variables': {}}'
2012-03-14 03:32:01,725 - ex2.log - DEBUG - Parsing doc path: '.' input_dir: '$variables' args : '{'$variables': {}}'
2012-03-14 03:32:01,725 - ex2.log - DEBUG - Finalizing dependencies between documents...
2012-03-14 03:32:01,725 - ex2.log - DEBUG - finalizing dependencies for hello.py
2012-03-14 03:32:01,725 - ex2.log - DEBUG - finalizing dependencies for hello.R
2012-03-14 03:32:01,725 - ex2.log - DEBUG - finalizing dependencies for hello.txt
2012-03-14 03:32:01,726 - ex2.log - DEBUG - Beginning topological sort...
2012-03-14 03:32:01,726 - ex2.log - DEBUG - Topological sort completed successfully.
We wrote our config file like this:
And this means we have listed hello.py and hello.R in two different places, once at the top level, and again when we listed them as explicit dependencies. As a shortcut, we can choose to just list them as dependencies, and Dexy will automatically create documents for them. The disadvantage of this approach is that we can't pass attributes to these documents, but when you don't need to pass extra attributes, this can save you some typing. Note: This only works for inputs in the same directory. If your inputs are in another directory, then you need to create them separately.
Here is the shorter version:
for doc in controller.docs:
s.write(doc.key())
s.write(" inputs: %s\n" % repr([d.key() for d in doc.inputs]))
hello.py inputs: [] hello.R inputs: [] hello.txt inputs: [u'hello.py', u'hello.R']
Here is what gets written to the Dexy log:
2012-03-14 03:32:01,727 - ex2a.log - DEBUG - project root excluded directories ignore, output-long, output, ../../../../artifacts, logs
2012-03-14 03:32:01,727 - ex2a.log - DEBUG - directories excluded at all levels .bzr, .hg, .git, .svn
2012-03-14 03:32:01,727 - ex2a.log - INFO - loading config file .dexy
2012-03-14 03:32:01,728 - ex2a.log - INFO - loading config file ./.dexy
2012-03-14 03:32:01,728 - ex2a.log - INFO - loading config file .dexy
2012-03-14 03:32:01,728 - ex2a.log - INFO - loading config file ./.dexy
2012-03-14 03:32:01,728 - ex2a.log - DEBUG - Parsing doc path: '.' input_dir: '$globals' args : '{'$variables': {}}'
2012-03-14 03:32:01,728 - ex2a.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.txt' args : '{u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}'
2012-03-14 03:32:01,728 - ex2a.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.py' args : '{}'
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - creating doc hello.py for glob hello.py
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - no existing key hello.py
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - updating with: '{}'
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - Doc args after update: '{}'
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.R' args : '{}'
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - creating doc hello.R for glob hello.R
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - no existing key hello.R
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - updating with: '{}'
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - Doc args after update: '{}'
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - creating doc hello.txt for glob hello.txt
2012-03-14 03:32:01,729 - ex2a.log - DEBUG - no existing key hello.txt
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - updating with: '{u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}'
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - Doc args after update: '{u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}'
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - Parsing doc path: '.' input_dir: '$variables' args : '{'$variables': {}}'
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - Finalizing dependencies between documents...
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - finalizing dependencies for hello.py
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - finalizing dependencies for hello.R
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - finalizing dependencies for hello.txt
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - Beginning topological sort...
2012-03-14 03:32:01,730 - ex2a.log - DEBUG - Topological sort completed successfully.
So far, all we've done is set Dexy up. We haven't actually run Dexy. Now, let's add a filter to our documents and actually run Dexy.
Here is our config:
'jinja' is a Filter Alias, a short key that we use to identify the filter we want to run. Each filter defines a list of its aliases (which must be unique) and Dexy figures out that when you put "|jinja" into your config file, it should run the JinjaFilter at that point. The "|" is the pipe symbol, the same one you type on the command line to pipe output from one command into another. You can apply as many filters as you want, the output from each filter becomes the input for the next filter in the chain.
At the beginning of each Dexy run, a list of the filter aliases and the filters they map to is populated. Here is what gets written to the log:
2012-03-14 03:32:01,735 - ex3.populate.filters.log - INFO - Automatically loading all dexy.filters found in /mnt/build-dexy-site/dexy/dexy/filters 2012-03-14 03:32:01,735 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/phantomjs_filters.py 2012-03-14 03:32:01,753 - ex3.populate.filters.log - INFO - registered alias phsecshot for class PhantomJsRenderJavascriptInteractiveFilter 2012-03-14 03:32:01,754 - ex3.populate.filters.log - INFO - registered alias phrender for class PhantomJsRenderSubprocessFilter 2012-03-14 03:32:01,754 - ex3.populate.filters.log - INFO - registered alias phantomjs for class PhantomJsStdoutFilter 2012-03-14 03:32:01,754 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/filename_filters.py 2012-03-14 03:32:01,754 - ex3.populate.filters.log - INFO - registered alias fn for class FilenameFilter 2012-03-14 03:32:01,755 - ex3.populate.filters.log - INFO - registered alias filenames for class FilenamesFilter 2012-03-14 03:32:01,755 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/clang_filters.py 2012-03-14 03:32:01,755 - ex3.populate.filters.log - INFO - registered alias cfussy for class CFussySubprocessCompileFilter 2012-03-14 03:32:01,755 - ex3.populate.filters.log - INFO - registered alias c for class CSubprocessCompileFilter 2012-03-14 03:32:01,755 - ex3.populate.filters.log - INFO - registered alias gcc for class CSubprocessCompileFilter 2012-03-14 03:32:01,756 - ex3.populate.filters.log - INFO - registered alias cinput for class CSubprocessCompileInputFilter 2012-03-14 03:32:01,756 - ex3.populate.filters.log - INFO - registered alias clang for class ClangSubprocessCompileFilter 2012-03-14 03:32:01,756 - ex3.populate.filters.log - INFO - registered alias clanginput for class ClangSubprocessCompileInputFilter 2012-03-14 03:32:01,756 - ex3.populate.filters.log - INFO - registered alias cpp for class CppSubprocessCompileFilter 2012-03-14 03:32:01,756 - ex3.populate.filters.log - INFO - registered alias cppinput for class CppSubprocessCompileInputFilter 2012-03-14 03:32:01,756 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/posterous_filters.py 2012-03-14 03:32:01,757 - ex3.populate.filters.log - WARNING - filters defined in dexy.filters.posterous_filters are not available: No module named blog_filter 2012-03-14 03:32:01,757 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/templating_filters.py 2012-03-14 03:32:01,758 - ex3.populate.filters.log - INFO - registered alias jinja for class JinjaFilter 2012-03-14 03:32:01,758 - ex3.populate.filters.log - INFO - registered alias jinjatext for class JinjaTextFilter 2012-03-14 03:32:01,758 - ex3.populate.filters.log - INFO - registered alias template for class TemplateFilter 2012-03-14 03:32:01,758 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/xdoc_filters.py 2012-03-14 03:32:01,784 - ex3.populate.filters.log - INFO - registered alias pydoc for class PythonDocumentationFilter 2012-03-14 03:32:01,784 - ex3.populate.filters.log - INFO - registered alias rdoc for class RDocumentationFilter 2012-03-14 03:32:01,784 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/markdown_filters.py 2012-03-14 03:32:01,826 - ex3.populate.filters.log - INFO - registered alias markdown for class MarkdownFilter 2012-03-14 03:32:01,826 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/confluence_filters.py 2012-03-14 03:32:01,829 - ex3.populate.filters.log - INFO - registered alias cfl for class ConfluenceFilter 2012-03-14 03:32:01,829 - ex3.populate.filters.log - INFO - registered alias confluence for class ConfluenceFilter 2012-03-14 03:32:01,829 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/mako_filters.py 2012-03-14 03:32:01,836 - ex3.populate.filters.log - INFO - registered alias mako for class MakoFilter 2012-03-14 03:32:01,836 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/pexpect_filters.py 2012-03-14 03:32:01,839 - ex3.populate.filters.log - INFO - registered alias clj for class ClojureInteractiveFilter 2012-03-14 03:32:01,839 - ex3.populate.filters.log - INFO - registered alias cljint for class ClojureInteractiveFilter 2012-03-14 03:32:01,839 - ex3.populate.filters.log - INFO - registered alias ipython for class IpythonPexpectReplFilter 2012-03-14 03:32:01,839 - ex3.populate.filters.log - INFO - registered alias shintp for class KshInteractiveFilter 2012-03-14 03:32:01,840 - ex3.populate.filters.log - INFO - registered alias shintpn for class KshInteractiveNumberedPromptFilter 2012-03-14 03:32:01,840 - ex3.populate.filters.log - INFO - registered alias shint for class KshInteractiveStrictFilter 2012-03-14 03:32:01,840 - ex3.populate.filters.log - INFO - registered alias pexpectreplfilter for class PexpectReplFilter 2012-03-14 03:32:01,840 - ex3.populate.filters.log - INFO - registered alias phpint for class PhpInteractiveFilter 2012-03-14 03:32:01,840 - ex3.populate.filters.log - INFO - registered alias pycon for class PythonPexpectReplFilter 2012-03-14 03:32:01,840 - ex3.populate.filters.log - INFO - registered alias pyrepl for class PythonPexpectReplFilter 2012-03-14 03:32:01,840 - ex3.populate.filters.log - INFO - registered alias r for class RPexpectReplFilter 2012-03-14 03:32:01,840 - ex3.populate.filters.log - INFO - registered alias rint for class RPexpectReplFilter 2012-03-14 03:32:01,841 - ex3.populate.filters.log - INFO - registered alias jsint for class RhinoInteractiveFilter 2012-03-14 03:32:01,841 - ex3.populate.filters.log - INFO - registered alias rhinoint for class RhinoInteractiveFilter 2012-03-14 03:32:01,841 - ex3.populate.filters.log - INFO - registered alias irb for class RubyPexpectReplFilter 2012-03-14 03:32:01,841 - ex3.populate.filters.log - INFO - registered alias rbrepl for class RubyPexpectReplFilter 2012-03-14 03:32:01,841 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/boto_filters.py 2012-03-14 03:32:01,841 - ex3.populate.filters.log - INFO - registered alias boto for class BotoUploadFilter 2012-03-14 03:32:01,841 - ex3.populate.filters.log - INFO - registered alias botoup for class BotoUploadFilter 2012-03-14 03:32:01,841 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/ansi2html_filters.py 2012-03-14 03:32:01,844 - ex3.populate.filters.log - INFO - registered alias ansi2html for class Ansi2HtmlFilter 2012-03-14 03:32:01,844 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/pytextile_filters.py 2012-03-14 03:32:01,844 - ex3.populate.filters.log - WARNING - filters defined in dexy.filters.pytextile_filters are not available: No module named textile 2012-03-14 03:32:01,844 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/templating_plugins.py 2012-03-14 03:32:01,844 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/cheetah_filters.py 2012-03-14 03:32:01,863 - ex3.populate.filters.log - INFO - registered alias cheetah for class CheetahFilter 2012-03-14 03:32:01,863 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/gdata_filters.py 2012-03-14 03:32:01,863 - ex3.populate.filters.log - WARNING - filters defined in dexy.filters.gdata_filters are not available: No module named gdata.photos.service 2012-03-14 03:32:01,863 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/archive_filters.py 2012-03-14 03:32:01,865 - ex3.populate.filters.log - INFO - registered alias archive for class ArchiveFilter 2012-03-14 03:32:01,865 - ex3.populate.filters.log - INFO - registered alias tgz for class ArchiveFilter 2012-03-14 03:32:01,865 - ex3.populate.filters.log - INFO - registered alias tgzdir for class UnprocessedDirectoryArchiveFilter 2012-03-14 03:32:01,865 - ex3.populate.filters.log - INFO - registered alias zip for class ZipArchiveFilter 2012-03-14 03:32:01,865 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/subprocess_filters.py 2012-03-14 03:32:01,866 - ex3.populate.filters.log - INFO - registered alias asciidoc for class AsciidocFilter 2012-03-14 03:32:01,866 - ex3.populate.filters.log - INFO - registered alias bib for class BibFilter 2012-03-14 03:32:01,867 - ex3.populate.filters.log - INFO - registered alias dot for class DotFilter 2012-03-14 03:32:01,867 - ex3.populate.filters.log - INFO - registered alias graphviz for class DotFilter 2012-03-14 03:32:01,867 - ex3.populate.filters.log - INFO - registered alias embedfonts for class EmbedFonts 2012-03-14 03:32:01,867 - ex3.populate.filters.log - INFO - registered alias prepress for class EmbedFonts 2012-03-14 03:32:01,867 - ex3.populate.filters.log - INFO - registered alias espeak for class EspeakFilter 2012-03-14 03:32:01,867 - ex3.populate.filters.log - INFO - registered alias fortran for class FortranFilter 2012-03-14 03:32:01,867 - ex3.populate.filters.log - INFO - registered alias htlatex for class HtLatexFilter 2012-03-14 03:32:01,867 - ex3.populate.filters.log - INFO - registered alias html2pdf for class Html2PdfSubprocessFilter 2012-03-14 03:32:01,868 - ex3.populate.filters.log - INFO - registered alias wkhtmltopdf for class Html2PdfSubprocessFilter 2012-03-14 03:32:01,868 - ex3.populate.filters.log - INFO - registered alias latex for class LatexFilter 2012-03-14 03:32:01,868 - ex3.populate.filters.log - INFO - registered alias pandoc for class PandocFilter 2012-03-14 03:32:01,868 - ex3.populate.filters.log - INFO - registered alias pdf2img for class Pdf2ImgSubprocessFilter 2012-03-14 03:32:01,868 - ex3.populate.filters.log - INFO - registered alias pdf2png for class Pdf2ImgSubprocessFilter 2012-03-14 03:32:01,868 - ex3.populate.filters.log - INFO - registered alias pdf2jpg for class Pdf2JpgSubprocessFilter 2012-03-14 03:32:01,869 - ex3.populate.filters.log - INFO - registered alias ps2pdf for class Ps2PdfSubprocessFilter 2012-03-14 03:32:01,869 - ex3.populate.filters.log - INFO - registered alias rintbatch for class RBatchFilter 2012-03-14 03:32:01,869 - ex3.populate.filters.log - INFO - registered alias rout for class ROutputBatchFilter 2012-03-14 03:32:01,869 - ex3.populate.filters.log - INFO - registered alias routbatch for class ROutputBatchFilter 2012-03-14 03:32:01,869 - ex3.populate.filters.log - INFO - registered alias rlrb for class RagelRubySubprocessFilter 2012-03-14 03:32:01,869 - ex3.populate.filters.log - INFO - registered alias ragelruby for class RagelRubySubprocessFilter 2012-03-14 03:32:01,869 - ex3.populate.filters.log - INFO - registered alias rd2pdf for class Rd2PdfFilter 2012-03-14 03:32:01,869 - ex3.populate.filters.log - INFO - registered alias Rd2pdf for class Rd2PdfFilter 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - registered alias tikz for class TikzPgfFilter 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/process_filters.py 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - registered alias processfilter for class ProcessFilter 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - registered alias subprocesscompilefilter for class SubprocessCompileFilter 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - registered alias subprocesscompileinputfilter for class SubprocessCompileInputFilter 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - registered alias subprocessfilter for class SubprocessFilter 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - registered alias subprocessstdoutfilter for class SubprocessStdoutFilter 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - registered alias subprocessstdoutinputfilefilter for class SubprocessStdoutInputFileFilter 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - registered alias subprocessstdoutinputfilter for class SubprocessStdoutInputFilter 2012-03-14 03:32:01,870 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/python_filters.py 2012-03-14 03:32:01,887 - ex3.populate.filters.log - INFO - registered alias easyhtml for class EasyHtml 2012-03-14 03:32:01,887 - ex3.populate.filters.log - INFO - registered alias ft for class FooterFilter 2012-03-14 03:32:01,887 - ex3.populate.filters.log - INFO - registered alias footer for class FooterFilter 2012-03-14 03:32:01,887 - ex3.populate.filters.log - INFO - registered alias head for class HeadFilter 2012-03-14 03:32:01,887 - ex3.populate.filters.log - INFO - registered alias hd for class HeaderFilter 2012-03-14 03:32:01,887 - ex3.populate.filters.log - INFO - registered alias header for class HeaderFilter 2012-03-14 03:32:01,887 - ex3.populate.filters.log - INFO - registered alias join for class JoinFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias tags for class MarkupTagsFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias ppjson for class PrettyPrintJsonFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias lines for class SectionsByLineFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias silly for class SillyFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias split for class SplitHtmlFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias splithtml for class SplitHtmlFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias splitlatex for class SplitLatexFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias ss for class StartSpaceFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias startspace for class StartSpaceFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias swagger for class SwaggerApiFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias test for class TestFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias ww for class WordWrapFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - registered alias wrap for class WordWrapFilter 2012-03-14 03:32:01,888 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/tumblr_filters.py 2012-03-14 03:32:01,889 - ex3.populate.filters.log - WARNING - filters defined in dexy.filters.tumblr_filters are not available: No module named tumblr 2012-03-14 03:32:01,889 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/nltk_filters.py 2012-03-14 03:32:02,152 - ex3.populate.filters.log - INFO - registered alias nltk for class NltkFilter 2012-03-14 03:32:02,153 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/idio_filters.py 2012-03-14 03:32:02,153 - ex3.populate.filters.log - INFO - registered alias idio for class IdioFilter 2012-03-14 03:32:02,153 - ex3.populate.filters.log - INFO - registered alias idiopidae for class IdioFilter 2012-03-14 03:32:02,153 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/misc_filters.py 2012-03-14 03:32:02,153 - ex3.populate.filters.log - INFO - registered alias cb for class ConvertBashExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias ch for class ConvertHTMLExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias cj for class ConvertJsonExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias ct for class ConvertTextExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias bn for class ForceBmpExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias forcebmp for class ForceBmpExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias gn for class ForceGifExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias forcegif for class ForceGifExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias h for class ForceHtmlExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias forcehtml for class ForceHtmlExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias jn for class ForceJpgExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias forcejpg for class ForceJpgExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias j for class ForceJsonExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias forcejson for class ForceJsonExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias l for class ForceLatexExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias forcelatex for class ForceLatexExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias p for class ForcePdfExtensionFilter 2012-03-14 03:32:02,154 - ex3.populate.filters.log - INFO - registered alias forcepdf for class ForcePdfExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - registered alias pn for class ForcePngExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - registered alias forcepng for class ForcePngExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - registered alias forcer for class ForceRExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - registered alias svg for class ForceSvgExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - registered alias forcesvg for class ForceSvgExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - registered alias t for class ForceTextExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - registered alias forcetext for class ForceTextExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - registered alias x for class ForceXmlExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - registered alias forcexml for class ForceXmlExtensionFilter 2012-03-14 03:32:02,155 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/tempdir_filters.py 2012-03-14 03:32:02,171 - ex3.populate.filters.log - INFO - registered alias shtmp for class KshTempdirInteractiveFilter 2012-03-14 03:32:02,171 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/java_filters.py 2012-03-14 03:32:02,171 - ex3.populate.filters.log - INFO - registered alias java for class JavaFilter 2012-03-14 03:32:02,172 - ex3.populate.filters.log - INFO - registered alias javac for class JavacFilter 2012-03-14 03:32:02,172 - ex3.populate.filters.log - INFO - registered alias javadoc for class JavadocsJsonFilter 2012-03-14 03:32:02,172 - ex3.populate.filters.log - INFO - registered alias javadocs for class JavadocsJsonFilter 2012-03-14 03:32:02,172 - ex3.populate.filters.log - INFO - registered alias jirb for class JirbFilter 2012-03-14 03:32:02,172 - ex3.populate.filters.log - INFO - registered alias jruby for class JrubyFilter 2012-03-14 03:32:02,172 - ex3.populate.filters.log - INFO - registered alias jython for class JythonFilter 2012-03-14 03:32:02,172 - ex3.populate.filters.log - INFO - registered alias jythoni for class JythonInteractiveFilter 2012-03-14 03:32:02,172 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/cucumber_filters.py 2012-03-14 03:32:02,173 - ex3.populate.filters.log - INFO - registered alias cuke for class CucumberFilter 2012-03-14 03:32:02,173 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/wordpress_filters.py 2012-03-14 03:32:02,173 - ex3.populate.filters.log - INFO - registered alias wp for class WordPressFilter 2012-03-14 03:32:02,173 - ex3.populate.filters.log - INFO - registered alias wordpress for class WordPressFilter 2012-03-14 03:32:02,173 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/mediawiki_filters.py 2012-03-14 03:32:02,173 - ex3.populate.filters.log - INFO - registered alias mediawiki for class MediaWikiFilter 2012-03-14 03:32:02,173 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/magick_filters.py 2012-03-14 03:32:02,188 - ex3.populate.filters.log - INFO - registered alias im-crop for class Imlib2Crop 2012-03-14 03:32:02,188 - ex3.populate.filters.log - INFO - registered alias im-flipv for class Imlib2Filter 2012-03-14 03:32:02,188 - ex3.populate.filters.log - INFO - registered alias im-fliph for class Imlib2FlipHorizontalFilter 2012-03-14 03:32:02,188 - ex3.populate.filters.log - INFO - registered alias im-grid for class Imlib2GridFilter 2012-03-14 03:32:02,188 - ex3.populate.filters.log - INFO - registered alias im-scale for class Imlib2ScaleFilter 2012-03-14 03:32:02,189 - ex3.populate.filters.log - INFO - registered alias im-thumb for class Imlib2Thumb 2012-03-14 03:32:02,189 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/pygments_filters.py 2012-03-14 03:32:02,192 - ex3.populate.filters.log - INFO - registered alias pyg for class PygmentsFilter 2012-03-14 03:32:02,192 - ex3.populate.filters.log - INFO - registered alias pygments for class PygmentsFilter 2012-03-14 03:32:02,192 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/api_filters.py 2012-03-14 03:32:02,192 - ex3.populate.filters.log - INFO - registered alias apis for class ApiFilter 2012-03-14 03:32:02,192 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/deprecated_filters.py 2012-03-14 03:32:02,192 - ex3.populate.filters.log - INFO - registered alias cp for class DeprecatedCopyFilter 2012-03-14 03:32:02,192 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/tenderapp_filters.py 2012-03-14 03:32:02,192 - ex3.populate.filters.log - INFO - registered alias tenderapp for class TenderappFilter 2012-03-14 03:32:02,193 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/xml_filters.py 2012-03-14 03:32:02,193 - ex3.populate.filters.log - WARNING - filters defined in dexy.filters.xml_filters are not available: No module named lxml 2012-03-14 03:32:02,193 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/pythonmagick_filters.py 2012-03-14 03:32:02,193 - ex3.populate.filters.log - WARNING - filters defined in dexy.filters.pythonmagick_filters are not available: No module named PythonMagick 2012-03-14 03:32:02,193 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/pynliner_filters.py 2012-03-14 03:32:02,446 - ex3.populate.filters.log - INFO - registered alias pynliner for class PynlinerFilter 2012-03-14 03:32:02,446 - ex3.populate.filters.log - INFO - registered alias inlinecss for class PynlinerFilter 2012-03-14 03:32:02,446 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/stdout_filters.py 2012-03-14 03:32:02,448 - ex3.populate.filters.log - INFO - registered alias sh for class BashSubprocessStdoutFilter 2012-03-14 03:32:02,448 - ex3.populate.filters.log - INFO - registered alias bash for class BashSubprocessStdoutFilter 2012-03-14 03:32:02,448 - ex3.populate.filters.log - INFO - registered alias shinput for class BashSubprocessStdoutInputFilter 2012-03-14 03:32:02,448 - ex3.populate.filters.log - INFO - registered alias clean for class CleanSubprocessStdoutFilter 2012-03-14 03:32:02,448 - ex3.populate.filters.log - INFO - registered alias strings for class CleanSubprocessStdoutFilter 2012-03-14 03:32:02,448 - ex3.populate.filters.log - INFO - registered alias cowsay for class CowsaySubprocessStdoutFilter 2012-03-14 03:32:02,448 - ex3.populate.filters.log - INFO - registered alias cowthink for class CowthinkSubprocessStdoutFilter 2012-03-14 03:32:02,449 - ex3.populate.filters.log - INFO - registered alias escript for class EscriptSubprocessStdoutFilter 2012-03-14 03:32:02,449 - ex3.populate.filters.log - INFO - registered alias irbout for class IrbSubprocessStdoutFilter 2012-03-14 03:32:02,449 - ex3.populate.filters.log - INFO - registered alias irboutinput for class IrbSubprocessStdoutInputFilter 2012-03-14 03:32:02,449 - ex3.populate.filters.log - INFO - registered alias lua for class LuaFilter 2012-03-14 03:32:02,449 - ex3.populate.filters.log - INFO - registered alias lynxdump for class LynxDumpFilter 2012-03-14 03:32:02,449 - ex3.populate.filters.log - INFO - registered alias man for class ManPageSubprocessStdoutFilter 2012-03-14 03:32:02,449 - ex3.populate.filters.log - INFO - registered alias zzzdoesnotexist for class NonexistentFilter 2012-03-14 03:32:02,450 - ex3.populate.filters.log - INFO - registered alias php for class PhpFilter 2012-03-14 03:32:02,450 - ex3.populate.filters.log - INFO - registered alias py for class PythonSubprocessStdoutFilter 2012-03-14 03:32:02,450 - ex3.populate.filters.log - INFO - registered alias pyout for class PythonSubprocessStdoutFilter 2012-03-14 03:32:02,450 - ex3.populate.filters.log - INFO - registered alias pyinput for class PythonSubprocessStdoutInputFilter 2012-03-14 03:32:02,450 - ex3.populate.filters.log - INFO - registered alias rlrbd for class RagelRubyDotFilter 2012-03-14 03:32:02,450 - ex3.populate.filters.log - INFO - registered alias ragelrubydot for class RagelRubyDotFilter 2012-03-14 03:32:02,450 - ex3.populate.filters.log - INFO - registered alias rdconv for class RdConvFilter 2012-03-14 03:32:02,450 - ex3.populate.filters.log - INFO - registered alias redcloth for class RedclothFilter 2012-03-14 03:32:02,450 - ex3.populate.filters.log - INFO - registered alias textile for class RedclothFilter 2012-03-14 03:32:02,451 - ex3.populate.filters.log - INFO - registered alias redclothl for class RedclothLatexFilter 2012-03-14 03:32:02,451 - ex3.populate.filters.log - INFO - registered alias latextile for class RedclothLatexFilter 2012-03-14 03:32:02,451 - ex3.populate.filters.log - INFO - registered alias regetron for class RegetronSubprocessStdoutInputFileFilter 2012-03-14 03:32:02,451 - ex3.populate.filters.log - INFO - registered alias js for class RhinoSubprocessStdoutFilter 2012-03-14 03:32:02,451 - ex3.populate.filters.log - INFO - registered alias rhino for class RhinoSubprocessStdoutFilter 2012-03-14 03:32:02,451 - ex3.populate.filters.log - INFO - class Rst2BeamerFilter is not available because rst2beamer not found 2012-03-14 03:32:02,451 - ex3.populate.filters.log - INFO - class Rst2HtmlFilter is not available because rst2html.py not found 2012-03-14 03:32:02,452 - ex3.populate.filters.log - INFO - class Rst2LatexFilter is not available because rst2latex.py not found 2012-03-14 03:32:02,452 - ex3.populate.filters.log - INFO - registered alias rb for class RubySubprocessStdoutFilter 2012-03-14 03:32:02,452 - ex3.populate.filters.log - INFO - registered alias rbinput for class RubySubprocessStdoutInputFilter 2012-03-14 03:32:02,452 - ex3.populate.filters.log - INFO - registered alias sed for class SedSubprocessStdoutInputFilter 2012-03-14 03:32:02,452 - ex3.populate.filters.log - INFO - registered alias sloc for class SloccountFilter 2012-03-14 03:32:02,452 - ex3.populate.filters.log - INFO - registered alias sloccount for class SloccountFilter 2012-03-14 03:32:02,452 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/vanillaforums_filters.py 2012-03-14 03:32:02,457 - ex3.populate.filters.log - INFO - registered alias vanillacomment for class VanillaForumCommentFilter 2012-03-14 03:32:02,457 - ex3.populate.filters.log - INFO - registered alias vanilla for class VanillaForumFilter 2012-03-14 03:32:02,457 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/wiki_filters.py 2012-03-14 03:32:02,457 - ex3.populate.filters.log - WARNING - filters defined in dexy.filters.wiki_filters are not available: No module named mwclient 2012-03-14 03:32:02,457 - ex3.populate.filters.log - INFO - Loading filters in /mnt/build-dexy-site/dexy/dexy/filters/blog_filters.py 2012-03-14 03:32:02,457 - ex3.populate.filters.log - INFO - registered alias blogfilter for class BlogFilter 2012-03-14 03:32:02,457 - ex3.populate.filters.log - INFO - ...finished loading filters from /mnt/build-dexy-site/dexy/dexy/filters
Sometimes, filters aren't available because they need software that isn't installed. A warning is written to the logfile in these cases. Dexy will raise an exception if you try to use a filter that isn't available.
The jinja filter will let us insert content into the hello.txt document. Let's take a look at that document now:
Here is some Python:
{{ d['hello.py'] }}
Here is some R:
{{ d['hello.R'] }}
The {{ and }} symbols are Jinja content tags, whatever goes inside those tags gets evaluated by Jinja and the value is put in that place. The 'd' variable is a dictionary which has the contents of all the Dexy inputs, referred to by their keys. So, we are asking for the contents of hello.py and hello.R to be inserted into the hello.txt template at those points. We have not applied any filters to hello.py or hello.R, so Dexy will insert the unchanged content of those files. We will try applying filters to these files in the next section.
We will run this by doing the same setup as before:
controller = Controller(args)
controller.log = log
controller.load_config()
controller.process_config()
controller.virtual_docs = []
and then adding
Here is what gets written to the log:
2012-03-14 03:32:01,732 - ex3.log - DEBUG - project root excluded directories ignore, output-long, output, ../../../../artifacts, logs
2012-03-14 03:32:01,732 - ex3.log - DEBUG - directories excluded at all levels .bzr, .hg, .git, .svn
2012-03-14 03:32:01,732 - ex3.log - INFO - loading config file .dexy
2012-03-14 03:32:01,732 - ex3.log - INFO - loading config file ./.dexy
2012-03-14 03:32:01,732 - ex3.log - INFO - loading config file .dexy
2012-03-14 03:32:01,733 - ex3.log - INFO - loading config file ./.dexy
2012-03-14 03:32:01,733 - ex3.log - DEBUG - Parsing doc path: '.' input_dir: '$globals' args : '{'$variables': {}}'
2012-03-14 03:32:01,733 - ex3.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.txt|jinja' args : '{u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}'
2012-03-14 03:32:01,733 - ex3.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.py' args : '{}'
2012-03-14 03:32:01,733 - ex3.log - DEBUG - creating doc hello.py for glob hello.py
2012-03-14 03:32:01,733 - ex3.log - DEBUG - no existing key hello.py
2012-03-14 03:32:01,733 - ex3.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,733 - ex3.log - DEBUG - updating with: '{}'
2012-03-14 03:32:01,733 - ex3.log - DEBUG - Doc args after update: '{}'
2012-03-14 03:32:01,733 - ex3.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.R' args : '{}'
2012-03-14 03:32:01,734 - ex3.log - DEBUG - creating doc hello.R for glob hello.R
2012-03-14 03:32:01,734 - ex3.log - DEBUG - no existing key hello.R
2012-03-14 03:32:01,734 - ex3.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,734 - ex3.log - DEBUG - updating with: '{}'
2012-03-14 03:32:01,734 - ex3.log - DEBUG - Doc args after update: '{}'
2012-03-14 03:32:01,734 - ex3.log - DEBUG - creating doc hello.txt|jinja for glob hello.txt
2012-03-14 03:32:01,734 - ex3.log - DEBUG - no existing key hello.txt|jinja
2012-03-14 03:32:01,734 - ex3.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:01,734 - ex3.log - DEBUG - updating with: '{u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}'
2012-03-14 03:32:01,734 - ex3.log - DEBUG - Doc args after update: '{u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}'
2012-03-14 03:32:01,734 - ex3.log - DEBUG - Parsing doc path: '.' input_dir: '$variables' args : '{'$variables': {}}'
2012-03-14 03:32:01,735 - ex3.log - DEBUG - Finalizing dependencies between documents...
2012-03-14 03:32:01,735 - ex3.log - DEBUG - finalizing dependencies for hello.py
2012-03-14 03:32:01,735 - ex3.log - DEBUG - finalizing dependencies for hello.R
2012-03-14 03:32:01,735 - ex3.log - DEBUG - finalizing dependencies for hello.txt|jinja
2012-03-14 03:32:01,735 - ex3.log - DEBUG - Beginning topological sort...
2012-03-14 03:32:01,735 - ex3.log - DEBUG - Topological sort completed successfully.
2012-03-14 03:32:02,468 - ex3.log - DEBUG - Setting artifact args for hello.py in hello.py to {}
2012-03-14 03:32:02,497 - ex3.log - INFO - (step 0) [run] hello.py -> 6f2665b58ea6948d9367f1a6adb1f1c9.py
2012-03-14 03:32:02,497 - ex3.log - DEBUG - Setting artifact args for hello.R in hello.R to {}
2012-03-14 03:32:02,497 - ex3.log - INFO - (step 0) [run] hello.R -> bd202fc40bc00960146cdcd977dc46d9.R
2012-03-14 03:32:02,498 - ex3.log - DEBUG - Setting artifact args for hello.txt in hello.txt|jinja to {u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}
2012-03-14 03:32:02,498 - ex3.log - INFO - (step 0) [run] hello.txt -> 1808d9ffb4a8f283744e5d728eeb1a1c.txt
2012-03-14 03:32:02,499 - ex3.log - DEBUG - Setting artifact args for hello.txt|jinja in hello.txt|jinja to {u'inputs': [u'hello.py', u'hello.R'], '$variables': {}}
2012-03-14 03:32:02,510 - ex3.log - DEBUG - Source code hash for JinjaFilter is ef3f7114bfdc8e7cd22f3c7e9de699ff
2012-03-14 03:32:02,510 - ex3.log - INFO - (starting step 1) [run] hello.txt|jinja -> cbecbac54638130ddf653f15ee93439e.txt
The log transcript starts out the same as before, but at the end we can see that Dexy is actually running the documents. For the first two documents, because they have no filters, Dexy only has to do Step 0, which is to just save the original file contents. For the third document, Dexy first does Step 0, then goes on to Step 1 which is running the first (and in this case, the only) filter.
Here is the result of running the hello.txt file through the jinja filter:
Here is some Python:
print "hello!"
Here is some R:
cat("Hello", "from", "R")
We can see that the jinja tags have been filled in with the contents of our input files.
Let's change our example now so that we can see the Python and R code, and also see the output that is generated by actually running the code. Here is our config file:
{
"hello.R": {},
"hello.R|rout": {},
"hello.py": {},
"hello.py|pyout": {},
"hello.txt|jinja": {
"allinputs": true
}
}
Now, the hello.py file is listed twice. Once with no filters, and once with the 'pyout' filter. Similarly, hello.R is listed once with no filters, and once with the 'rout' filter. The 'pyout' filter takes Python code and runs it through the Python interpreter, returning whatever gets written to STDOUT. The 'rout' filter does the same thing, but for R code instead of Python code.
Another change we've made is that, instead of listing each of these as inputs to hello.txt|jinja, we have taken a shortcut. Setting the "allinputs" attribute to "true" tells Dexy that every other document should be a dependency.
Here is the hello.txt file now:
Here is some Python:
{{ d['hello.py'] }}
Here is the output from running this Python code:
{{ d['hello.py|pyout'] }}
Here is some R:
{{ d['hello.R'] }}
Here is the output from running this R code:
{{ d['hello.R|rout'] }}
Here is the members ordered dict:
hello.R : <dexy.document.Document object at 0x3f39e50> hello.py|pyout : <dexy.document.Document object at 0x3f3c150> hello.py : <dexy.document.Document object at 0x3f3c310> hello.txt|jinja : <dexy.document.Document object at 0x3f3c390> hello.R|rout : <dexy.document.Document object at 0x3f3c550>
And here is the list of dependencies:
[(0, 3), (1, 3), (2, 3), (4, 3)]
Here is the ordering:
[0, 1, 2, 4, 3]
Using that ordering, the final list of documents in run order is:
for doc in controller.docs:
s.write(doc.key())
s.write(" inputs: %s\n" % repr([d.key() for d in doc.inputs]))
hello.R inputs: [] hello.py|pyout inputs: [] hello.py inputs: [] hello.R|rout inputs: [] hello.txt|jinja inputs: [u'hello.R', u'hello.py|pyout', u'hello.py', u'hello.R|rout']
Here is what gets written to the log file during the run:
2012-03-14 03:32:02,554 - ex4.log - DEBUG - project root excluded directories ignore, output-long, output, ../../../../artifacts, logs
2012-03-14 03:32:02,554 - ex4.log - DEBUG - directories excluded at all levels .bzr, .hg, .git, .svn
2012-03-14 03:32:02,554 - ex4.log - INFO - loading config file .dexy
2012-03-14 03:32:02,554 - ex4.log - INFO - loading config file ./.dexy
2012-03-14 03:32:02,554 - ex4.log - INFO - loading config file .dexy
2012-03-14 03:32:02,555 - ex4.log - INFO - loading config file ./.dexy
2012-03-14 03:32:02,555 - ex4.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.R' args : '{'$variables': {}}'
2012-03-14 03:32:02,555 - ex4.log - DEBUG - creating doc hello.R for glob hello.R
2012-03-14 03:32:02,555 - ex4.log - DEBUG - no existing key hello.R
2012-03-14 03:32:02,555 - ex4.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:02,555 - ex4.log - DEBUG - updating with: '{'$variables': {}}'
2012-03-14 03:32:02,555 - ex4.log - DEBUG - Doc args after update: '{'$variables': {}}'
2012-03-14 03:32:02,556 - ex4.log - DEBUG - Parsing doc path: '.' input_dir: '$variables' args : '{'$variables': {}}'
2012-03-14 03:32:02,556 - ex4.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.py|pyout' args : '{'$variables': {}}'
2012-03-14 03:32:02,556 - ex4.log - DEBUG - creating doc hello.py|pyout for glob hello.py
2012-03-14 03:32:02,556 - ex4.log - DEBUG - no existing key hello.py|pyout
2012-03-14 03:32:02,556 - ex4.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:02,556 - ex4.log - DEBUG - updating with: '{'$variables': {}}'
2012-03-14 03:32:02,556 - ex4.log - DEBUG - Doc args after update: '{'$variables': {}}'
2012-03-14 03:32:02,556 - ex4.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.py' args : '{'$variables': {}}'
2012-03-14 03:32:02,557 - ex4.log - DEBUG - creating doc hello.py for glob hello.py
2012-03-14 03:32:02,557 - ex4.log - DEBUG - no existing key hello.py
2012-03-14 03:32:02,557 - ex4.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:02,557 - ex4.log - DEBUG - updating with: '{'$variables': {}}'
2012-03-14 03:32:02,557 - ex4.log - DEBUG - Doc args after update: '{'$variables': {}}'
2012-03-14 03:32:02,557 - ex4.log - DEBUG - Parsing doc path: '.' input_dir: '$globals' args : '{'$variables': {}}'
2012-03-14 03:32:02,557 - ex4.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.txt|jinja' args : '{u'allinputs': True, '$variables': {}}'
2012-03-14 03:32:02,557 - ex4.log - DEBUG - creating doc hello.txt|jinja for glob hello.txt
2012-03-14 03:32:02,557 - ex4.log - DEBUG - no existing key hello.txt|jinja
2012-03-14 03:32:02,557 - ex4.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:02,558 - ex4.log - DEBUG - updating with: '{u'allinputs': True, '$variables': {}}'
2012-03-14 03:32:02,558 - ex4.log - DEBUG - Doc args after update: '{u'allinputs': True, '$variables': {}}'
2012-03-14 03:32:02,558 - ex4.log - DEBUG - Parsing doc path: '.' input_dir: 'hello.R|rout' args : '{'$variables': {}}'
2012-03-14 03:32:02,558 - ex4.log - DEBUG - creating doc hello.R|rout for glob hello.R
2012-03-14 03:32:02,558 - ex4.log - DEBUG - no existing key hello.R|rout
2012-03-14 03:32:02,558 - ex4.log - DEBUG - Doc args: '{}'
2012-03-14 03:32:02,558 - ex4.log - DEBUG - updating with: '{'$variables': {}}'
2012-03-14 03:32:02,558 - ex4.log - DEBUG - Doc args after update: '{'$variables': {}}'
2012-03-14 03:32:02,558 - ex4.log - DEBUG - Finalizing dependencies between documents...
2012-03-14 03:32:02,558 - ex4.log - DEBUG - finalizing dependencies for hello.R
2012-03-14 03:32:02,558 - ex4.log - DEBUG - finalizing dependencies for hello.py|pyout
2012-03-14 03:32:02,558 - ex4.log - DEBUG - finalizing dependencies for hello.py
2012-03-14 03:32:02,559 - ex4.log - DEBUG - finalizing dependencies for hello.txt|jinja
2012-03-14 03:32:02,559 - ex4.log - DEBUG - finalizing dependencies for hello.R|rout
2012-03-14 03:32:02,559 - ex4.log - DEBUG - Beginning topological sort...
2012-03-14 03:32:02,559 - ex4.log - DEBUG - Topological sort completed successfully.
2012-03-14 03:32:02,584 - ex4.log - DEBUG - Setting artifact args for hello.R in hello.R to {'$variables': {}}
2012-03-14 03:32:02,585 - ex4.log - INFO - (step 0) [run] hello.R -> 27e73506cfafe3a82bb3133ff241f6e2.R
2012-03-14 03:32:02,585 - ex4.log - DEBUG - Setting artifact args for hello.py in hello.py|pyout to {'$variables': {}}
2012-03-14 03:32:02,586 - ex4.log - INFO - (step 0) [run] hello.py -> 15e4ab280748427cc177074c598ff195.py
2012-03-14 03:32:02,586 - ex4.log - DEBUG - Setting artifact args for hello.py in hello.py to {'$variables': {}}
2012-03-14 03:32:02,587 - ex4.log - INFO - (step 0) [run] hello.py -> fc545854291a4e4341d2c3d2be9a4ef8.py
2012-03-14 03:32:02,587 - ex4.log - DEBUG - Setting artifact args for hello.R in hello.R|rout to {'$variables': {}}
2012-03-14 03:32:02,588 - ex4.log - INFO - (step 0) [run] hello.R -> 455e760ddd998d345da12804be5c0741.R
2012-03-14 03:32:02,588 - ex4.log - DEBUG - Setting artifact args for hello.txt in hello.txt|jinja to {u'allinputs': True, '$variables': {}}
2012-03-14 03:32:02,589 - ex4.log - INFO - (step 0) [run] hello.txt -> 899f2c2d4ab6b17e2bd319c6a62d76ac.txt
2012-03-14 03:32:02,589 - ex4.log - DEBUG - Setting artifact args for hello.py|pyout in hello.py|pyout to {'$variables': {}}
2012-03-14 03:32:02,600 - ex4.log - DEBUG - Source code hash for PythonSubprocessStdoutFilter is 52f9103e2b58814d838f058977fc3d26
2012-03-14 03:32:02,637 - ex4.log - INFO - (starting step 1) [run] hello.py|pyout -> 794ad0f6e807f891981ca3547908c16c.txt
2012-03-14 03:32:02,637 - ex4.log - DEBUG - about to run 'python 15e4ab280748427cc177074c598ff195.py ' in ../../../../artifacts
2012-03-14 03:32:02,689 - ex4.log - DEBUG - Setting artifact args for hello.R|rout in hello.R|rout to {'$variables': {}}
2012-03-14 03:32:02,700 - ex4.log - DEBUG - Source code hash for ROutputBatchFilter is f8175401cad88bcac99385e3c692b90f
2012-03-14 03:32:02,739 - ex4.log - INFO - (starting step 1) [run] hello.R|rout -> 17e811625ef1e0878a0a5731f9b22ade.txt
2012-03-14 03:32:02,740 - ex4.log - DEBUG - about to run 'R CMD BATCH --vanilla --quiet --slave --no-timing 455e760ddd998d345da12804be5c0741.R 17e811625ef1e0878a0a5731f9b22ade.txt' in ../../../../artifacts
2012-03-14 03:32:03,045 - ex4.log - DEBUG - Setting artifact args for hello.txt|jinja in hello.txt|jinja to {u'allinputs': True, '$variables': {}}
2012-03-14 03:32:03,046 - ex4.log - INFO - (starting step 1) [run] hello.txt|jinja -> 58aa73414fbec4e3a6701f3783e463d1.txt
And, here is the result of running hello.txt through the jinja filter:
Here is some Python:
print "hello!"
Here is the output from running this Python code:
hello!
Here is some R:
cat("Hello", "from", "R")
Here is the output from running this R code:
Hello from R
That's all for this chapter. Check out the filter references to see what filters are available.
This website was generated by Dexy. | This Page's Source | This Page's Log (large HTML page) | Back to Top
Content © 2011 Dr. Ana Nelson | Site Design © Copyright 2011 Andre Gagnon | All Rights Reserved.