🍕 Easy Configuration(ezkfg)

GitHub: 🍕 Easy Configuration(ezkfg)

I want to talk about a small project I recently wrote, ezkfg.

Why this name? Mainly because names like (kfg, config, cfg, ezcfg, ezconfig) were already taken

Secondly, configuration files are always in high demand; I use them in many projects but never found one I truly liked. Rather than keep searching, I decided to build my own wheel, taking the best parts and discarding the rest.

Come quickly to pip install ezkfg!

Several Configuration Methods

  • argparse:

    The most common is the argparse approach, which supports many data types, allows setting default values, supports setting optional values, and even supports descriptions for optional values. This is also the method I frequently use in many projects, especially in deep learning projects, because it allows for very convenient specification of experimental parameters. However, parameters are hard to persist; you either have to manually write file reading code or pass a list in the file first for parameter parsing.

  • config.py:

    This is probably my favorite and most commonly used form besides the above, because it is native Python content. You can define multiple config classes and import them via from config import config_xxx as config, which is quite convenient for configuration while supporting all Python data types. However, since it requires modifying import package information, changing parameters might require editing at least two files each time, which can be confusing.

  • YAML:

    YAML is also a very flexible configuration file type; it has a simple structure, can be loaded from files, is very convenient for reproducibility, and supports data structures like lists. It is a configuration file type I really like.

  • json:

    Same as above, but the file structure is more complex, and it becomes less intuitive once the content grows.

  • ini:

    Convenient for selecting multiple configurations, with built-in library support, but it lacks data types; everything is read as strings, requiring manual data conversion. Additionally, it is in k-v form, so it has no hierarchy.

At the same time, I also came across the addict project, where nested . attribute access is really quite convenient, but it lacks file I/O.

Clarifying Requirements

So, we still need to clarify the requirements.

The file is just a carrier; I personally believe that differences in file formats are all acceptable, perhaps with slight variations depending on the application scenario. For example, if I want to send my experimental content to others or quickly reproduce it on another machine, using a configuration file is clearly far superior to other methods.

Additionally, the form of invocation in experiments is also very important. First is pre-loading parameters. During pre-loading, some parameters need default values, while others might obviously not be used; in such cases, it is best not to occupy storage and to simplify the weight of the configuration file.

Second, data types are necessary, since parameters in experiments can take various forms.

Then there is parameter reference in the code. For specific parameters, if I want to achieve hierarchical access using nested . instead of calling with a long string of []. However, in other scenarios, such as two methods that share the same hyperparameters, using . for invocation is not conducive to code reuse. A better approach would be the config['method'].alpha form, which looks more intuitive; this way we can dynamically modify method to use the corresponding alpha values. Therefore, both forms are necessary.

Finally, there is file I/O, which must be supported. If you need to deploy elsewhere, obviously using a configuration file directly without modifying any original project code yields much better results.

Implementation

First and foremost are attribute access and index access; both are mandatory and must support nesting. Index access via [] is nothing special; it is a built-in feature because every class in Python actually has a __dict__ variable that stores all attributes of the class. All configuration parameters also borrow this dictionary for storage.

First is initialization. Currently, it supports several initialization methods: dictionaries, lists, and Config classes. dict requires a recursive solution, while Config can simply use the built-in update method.

Among these, __getitem__ and __setitem__ methods are implemented by calling __getattr__ and __setattr__ respectively, which is straightforward without ambiguity. The __setitem__ method requires splitting first, then recursing. However, the implementation of __getattr__ is also simple, just recursive querying. But __setattr__ is different; before setting, it calls the get method to query first, and if it does not exist

Finally, solve the file I/O problem.

Regarding handler, a registry is provided for users who like customization, allowing them to directly register custom handler into the package, or to parse custom file formats using existing handler.

Roadmap

The project has currently reached the v0.0.1 version, having gone through 3 pre-release versions.

However, there are some issues that are visibly apparent.

First, type conversion is not yet supported; default parameters can only be achieved by pre-loading a default file or by pre-converting argparse. The interfaces for py and ini files still need optimization, as the names are currently hard-coded.

Takeaways

To be honest, this project has indeed taught me a lot about various configuration file formats and their characteristics, as well as Python’s built-in functions and their call order. For example, __setattr__ calls __getattr__ first, but this is not implemented within __setattr__; instead, it is implemented at a higher level, so proper handling is required.

Besides that, the more important thing is to make it convenient for me to write configuration files for my own projects, since it’s likely that no one else will use it, hhh