Base class for filters which post content to a remote API. This class provides standard formats and locations for storing configuration and authentication information. Need to read config for the API in general, such as the base URL and an API key for authentication. Also need to read config for the particular task/document being uploaded. This could be stored in the dexy config entry, but this has two drawbacks. First, it makes it difficult to bulk define documents according to a pattern. Secondly, it makes it very difficult for dexy to modify this configuration to add additional information, such as the returned id for a newly created document. So, it is preferable to define a local file (defined by a relative path to the document in question) which can be overridden per-document in which we just store the API-related config and which can be modified by dexy without concern about identifying the entry in a .dexy file or accidentally overwriting some unrelated information.
apis
.*
.*
Setting | Description | Default |
---|---|---|
add-new-files | Boolean or list of extensions/patterns to match. | False |
added-in-version | Dexy version when this filter was first available. | |
additional-doc-filters | Filters to apply to additional documents created as side effects. | {} |
additional-doc-settings | Settings to apply to additional documents created as side effects. | {} |
api-key-name | The name of this API | None |
api-password | The password to sign into the API with. | None |
api-url | The url of the API endpoint. | None |
api-username | The username to sign into the API with. | None |
data-type | Alias of custom data class to use to store filter output. | generic |
document-api-config-file | Filename to store config for a file (can only have 1 per directory, dexy looks for suffix format first. | None |
document-api-config-postfix | Suffix to attach to content filename to indicate this is the config for that file. | -config.json |
examples | Templates which should be used as examples for this filter. | [] |
exclude-add-new-files | List of patterns to skip even if they match add-new-files. | [] |
exclude-new-files-from-dir | List of directories to skip when adding new files. | [] |
ext | File extension to output. | None |
extension-map | Dictionary mapping input extensions to default output extensions. | None |
help | Helpstring for plugin. | Base class for filters which post content to a remote API. This class provides standard formats and locations for storing configuration and authentication information. Need to read config for the API in general, such as the base URL and an API key for authentication. Also need to read config for the particular task/document being uploaded. This could be stored in the dexy config entry, but this has two drawbacks. First, it makes it difficult to bulk define documents according to a pattern. Secondly, it makes it very difficult for dexy to modify this configuration to add additional information, such as the returned id for a newly created document. So, it is preferable to define a local file (defined by a relative path to the document in question) which can be overridden per-document in which we just store the API-related config and which can be modified by dexy without concern about identifying the entry in a .dexy file or accidentally overwriting some unrelated information. |
input-extensions | List of extensions which this filter can accept as input. | ['.*'] |
keep-originals | Whether, if additional-doc-filters are specified, the original unmodified docs should also be added. | False |
master-api-key-file | Master API key file for user. | ~/.dexyapis |
mkdir | A directory which should be created in working dir. | None |
mkdirs | A list of directories which should be created in working dir. | [] |
nodoc | Whether filter should be excluded from documentation. | False |
output | Whether to output results of this filter by default by reporters such as 'output' or 'website'. | False |
output-extensions | List of extensions which this filter can produce as output. | ['.*'] |
override-workspace-exclude-filters | If True, document will be populated to other workspaces ignoring workspace-exclude-filters. | False |
preserve-prior-data-class | Whether output data class should be set to match the input data class. | False |
project-api-key-file | API key file for project. | .dexyapis |
require-output | Should dexy raise an exception if no output is produced by this filter? | True |
tags | Tags which describe the filter. | [] |
variables | A dictionary of variable names and values to make available to this filter. | {} |
vars | A dictionary of variable names and values to make available to this filter. | {} |
workspace-exclude-filters | Filters whose output should be excluded from workspace. | ['pyg'] |
workspace-includes | If set to a list of filenames or extensions, only these will be populated to working dir. | None |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | class ApiFilter(dexy.filter.DexyFilter):
"""
Base class for filters which post content to a remote API.
This class provides standard formats and locations for storing
configuration and authentication information.
Need to read config for the API in general, such as the base URL and an API
key for authentication.
Also need to read config for the particular task/document being uploaded.
This could be stored in the dexy config entry, but this has two drawbacks.
First, it makes it difficult to bulk define documents according to a
pattern. Secondly, it makes it very difficult for dexy to modify this
configuration to add additional information, such as the returned id for a
newly created document.
So, it is preferable to define a local file (defined by a relative path to
the document in question) which can be overridden per-document in which we
just store the API-related config and which can be modified by dexy without
concern about identifying the entry in a .dexy file or accidentally
overwriting some unrelated information.
"""
aliases = ['apis']
_settings = {
# Files to hold collections of API keys
'master-api-key-file' : ("Master API key file for user.", "~/.dexyapis"),
'project-api-key-file' : ("API key file for project.", ".dexyapis"),
# Parameters to be stored in collection files
'api-username' : ("The username to sign into the API with.", None),
'api-password' : ("The password to sign into the API with.", None),
'api-url' : ("The url of the API endpoint.", None),
# Files to hold info about a single document
'document-api-config-file' : ('Filename to store config for a file (can only have 1 per directory, dexy looks for suffix format first.', None),
'document-api-config-postfix' : ('Suffix to attach to content filename to indicate this is the config for that file.', '-config.json'),
'api-key-name' : ("The name of this API", None),
}
def api_key_locations(self):
return [self.setting('project-api-key-file'), self.setting('master-api-key-file')]
def docmd_create_keyfile(self):
return self.create_keyfile('master-api-key-file')
def create_keyfile(self, keyfilekey):
"""
Creates a key file.
"""
key_filename = os.path.expanduser(self.setting(keyfilekey))
if file_exists(key_filename):
raise dexy.exceptions.UserFeedback("File %s already exists!" % key_filename)
keyfile_content = {}
for filter_instance in dexy.filter.Filter:
if isinstance(filter_instance, ApiFilter) and not filter_instance.__class__ == ApiFilter:
api_key_name = filter_instance.setting('api-key-name')
# We want to create a keyfile for this filter instance.
keyfile_content[api_key_name] = {}
# Get all the entries we want in the keyfile.
for k, v in filter_instance.setting_values().items():
if k.startswith("api_"):
keyfile_content[api_key_name][k.replace("api_", "")] = "TODO"
with open(key_filename, "w") as f:
json.dump(keyfile_content, f, sort_keys = True, indent=4)
def document_config_file(self):
postfix_config_filename = "%s%s" % (os.path.splitext(self.output_data.name)[0], self.setting('document-api-config-postfix'))
if file_exists(postfix_config_filename):
return postfix_config_filename
else:
return os.path.join(self.output_data.parent_dir(), self.setting('document-api-config-file'))
def read_document_config(self):
document_config = self.document_config_file()
if file_exists(document_config):
with open(document_config, "r") as f:
return json.load(f)
else:
msg = "Filter %s needs a file %s, couldn't find it."
raise dexy.exceptions.UserFeedback(msg % (self.alias, document_config))
def save_document_config(self, config):
document_config = self.document_config_file()
with open(document_config, "w") as f:
json.dump(config, f, sort_keys=True, indent=4)
def read_param(self, param_name, default=False):
param_value = None
for filename in self.api_key_locations():
if "~" in filename:
filename = os.path.expanduser(filename)
if file_exists(filename):
with open(filename, "r") as f:
params = json.load(f)
if self.setting('api-key-name') in params:
param_value = params[self.setting('api-key-name')].get(param_name)
if param_value:
break
if param_value and isinstance(param_value, str) and param_value.startswith("$"):
# need to get value of bash variable
param_value_from_env = os.getenv(param_value.lstrip("$"))
if not param_value_from_env:
raise KeyError("Bash variable %s not defined in this environment!" % param_value)
param_value = param_value_from_env
if param_value:
return param_value
elif default:
return default
else:
msg = "Could not find %s for %s in: %s" % (param_name, self.setting('api-key-name'), ", ".join(self.api_key_locations()))
raise KeyError(msg)
|
Content © 2020 Dr. Ana Nelson | Site Design © Copyright 2011 Andre Gagnon | All Rights Reserved.