Configuration

The configuration is loaded by a ConfigurationManager that takes care of loading and merging included configuration files.

Configuration is loaded as part of the piped startup process.

What is a configuration file?

A configuration file is a file that contains a YAML mapping. A YAML mapping is essentially a Python dict that may contain any other Python primitive or object.

Example valid configuration file:

pipelines:
    ...

Invalid configuration files:

[1,2,3]

- 1
- 2
- 3

strings/numbers

An empty configuration file is allowed and is treated as an empty dict.

Note

Even though loading a YAML file can construct any object by using different constructors, providers usually only use primitive types, such as dict, list, tuple, str, int, float and so on.

Additional YAML constructors

In YAML, constructors are responsible for creating Python objects for the nodes in the configuration file. Constructors are referenced by tagging the YAML node with !constructor_name.

The following sections describe the built-in constructors used in piped:

!alias constructor

The !alias constructor can be used in configuration files to reference other parts of the configuration, even across multiple YAML files. The constructor supports specifying default values, which will be overridden by the aliased configuration key. Referencing non-dict values are currently not supported.

Example usage:

foo:
    bar:
        baz: [1,2,3]
        number: 42

zip: !alias:foo.bar
    number: 10
    zap: 93

In the above case, zip would end up with the following value:

baz: [1,2,3]
number: 42
zap: 93

!path constructor

The !path constructor is used in processor configurations to denote that a specific value is found in the baton.

Example usage:

pipelines:

my_pipeline:

bind-solr-client:

url: !path url

In the above example, the actual url used by the processor will be the url attribute or item of each baton.

Note

Processors that don’t explicitly support this constructor will treat the value as a string, without the !path part. Refer to each processors documentation for more detailed information.

!pipedpath constructor

Constructs twisted.python.filepath.FilePath instances relative to the Piped source root path. This is used to reference bundled data and configuration files.

Example that uses !pipedpath to include the default built-in configuration file:

includes:
    - !pipedpath conf.yml

The contents of a configuration file

The configuration is used by providers and processors, which are loaded as plugins when Piped is started.

See the overview of available providers for an overview over root configuration keys that have special meaning.

Including other configuration files

Includes are specified under the top level key includes in the configuration files.

includes:
    - path_string
    - /absolute_path_string
    - ~/using/a/home/folder
    - $VAR/using/an/environment/variable
    - prefix.path: ./relative_file/to_include

Included configuration files may specify their own includes, which are loaded recursively.

If the include is a dict, as in the prefix.path example above, the key, value pairs are interpreted as prefix, path pairs. When they are resolved, the included configuration is not merged with the root of the current configuration, but at the given prefix. For example:

Consider a configuration: config.yaml:

includes:
    - foo.bar: file.yaml

foo:
    bar:
        question: the question
    baz: 93

If file.yaml contains the following:

question: a question
answer: 42

Loading config.yaml would result in the following configuration:

foo:
    bar:
        question: the question
        answer: 42
    baz: 93

Merging

When a configuration file is loaded, all included configuration files are loaded and merged into the resulting configuration with the following precedence rules:

• The current configuration takes precedence before any included configuration.
• Included files are loaded in the list-order, and the latest configuration files takes precedence over the previously included files.

Examples

Given the following configuration files:

foo.yaml:

includes:
    - bar.yaml

data:
    foo: 42
    key: foo_value

bar.yaml:

data:
    bar: 93
    key: bar_value

Loading foo.yaml results in the following configuration:

data:
    bar: 93
    foo: 42
    key: foo_value

Loops

If there are loops in the include directives, the last include directive encountered that causes the loop prints a warning and is skipped.

Consider these two configuration files:

foo.yaml:

includes:
    - bar.yaml

foo: foo
number: 42

bar.yaml:

includes:
    - foo.yaml

bar: bar
number: 93

Loading foo.yaml results in the following configuration:

bar: bar
foo: foo
number: 42

While loading bar.yaml results in the following configuration:

foo: foo
bar: bar
number: 93

Runtime changes

Some providers and processors provide runtime access to the configuration. While the configuration is not marked read-only, runtime changes are not supported, and may cause undesirable or undetermined behaviour, depending on how the provider or processor accesses the configuration values, as the provider may have copied the configuration file in order to perform some inline changes to it.

Instead, if you need to change the configuration of a running Piped instance, consider doing it via the specific provider or processor you want to configure.