ThreadSafeDatabaseMetadata

CuneiFox stores data for different companies in different file paths. It is essential that the program is able to distinguish and choose correct database files to work on. This is not an issue if the client-side request is processed one-by-one and no internal processes dealing with any of the companies' data are ever needed. However, these conditions severely limit the functionality and latency of the program. Hence this modified Metadata class is developed.

Main Idea & Origin

The core essential of ThreadSafeDatabaseMetadata is to somehow map each request to its proper database files, as well as initiate and teardown the database connections without affecting other requests running in parallel.

CuneiFox's flavour of ThreadSafeDatabaseMetadata is derived from what written under Thread-Safety and Multiple Databases in Peewee 3.14.4 Documentation. (The documentation for this version does not exist on the main channel anymore. The link points to a copied version hosted on another site. The essential content is also copied here for safekeeping.)

# Content from Peewee 3.14.4 Documentation

import threading
from peewee import Metadata, Model
class ThreadSafeDatabaseMetadata(Metadata):
    def __init__(self, *args, **kwargs):
        # database attribute is stored in a thread-local.
        self._local = threading.local()
        super(ThreadSafeDatabaseMetadata, self).__init__(*args, **kwargs)
    def _get_db(self):
        return getattr(self._local, 'database', self._database)
    def _set_db(self, db):
        self._local.database = self._database = db
    database = property(_get_db, _set_db)
class BaseModel(Model):
    class Meta:
        # Instruct peewee to use our thread-safe metadata implementation.
        model_metadata_class = ThreadSafeDatabaseMetadata

In the source material, the Metadata class solves this problem by storing the database information in a thread-local. This thread-based solution does not work for CuneiFox which requires a storage that is specific to:

• Each client-side request with immediate return
• Each server-side initiated thread (using pseudo-sessions)
• Each client-side initiated thread (long processes, e.g. reports, multi-document printing)

Storage Format

To achieve specific storages as outlined in the previous section, CuneiFox employs a dictionary named 'db_dict' stored as an attribute of the main app object.

The key for each items is a 2-tuple:

• For client-side requests: (str session_token, str request_id)
• For server-side threads: (str thread_name, None)
• For client-side threads: (str session_token, str thread_name)
• Fallback key: ('universal', None)

The reusable function 'establish_key' determines the case and key value for each database-related process.

The value of each dict item is yet another dict with Model ID as the key and the database object (the path to the file is embedded within) as the value.

app.db_dict = {tuple key0: {int model_id00: peewee.Database database_object00,
                            int model_id01: peewee.Database database_object01,
                            ...},
               tuple key1: {int model_id10: peewee.Database database_object10,
                            int model_id11: peewee.Database database_object11,
                            ...},
               ...}

Storage Life

The key-value pair within db_dict is initiated for each request/thread and teardown (popped) once the request/thread finished as detailed below. This mean, during operation, the dict only holds values for actively running requests/threads.

• For client-side requests: This group of items is very short-lived.
  • An item is initiated upon each client-side request as a part of Flask's 'before_request' routine.
  • The item is teardown as a part of Flask's 'teardown_request' routine.
• For server-side threads: This group is longer-lived, but the initiation and teardown are both parts of CuneiFox automatic routines.
  • An item is initiated as a part of 'establish_key' function when CuneiFox notices the thread is running as a pseudosession.
  • The item is deleted as a closing action of the pseudosessions.
• For client-side threads: This group requires some care on the developer's part. CuneiFox assumes all long processes requested from the clients to use CuneiFox native task tracking routine. Thus, the initiation and teardown of such db_dict items are tied to said routine.
  • An item is initiated as a part of 'init_task_db'.
  • The item is popped when the thread update its own status to either 'complete' or 'killed' via 'update_task_db'.

Key Functions

__init__

The main role of this function is to create a Metadata object associated with each specific model. It was found during testing that the function is only run once for each model when it is first encountered.

There are 2 notable attributes to describe here:

• is_static (bool): This is a new attribute introduced in CuneiFox. It shows whether the associated CuneiModel is tied with a statically defined database path or not. As elaborated further under the main article, this property is determined by the path embedded in the database object declared under the model's class Meta section.
  • A CuneiModel with static database path must have the proper path readily declared.
  • A CuneiModel with dynamic database path must have the path set to None.
• _database (peewee.Database): Stores the database object for future reference. This attribute is only applicable when is_static is True.

_get_db

This function grabs the proper database object for other parts of the program to work on. It is quite simple.

• If is_static is True, just return whatever _database is.
• Otherwise, look into app.db_dict and return the proper object.

_set_db

This function sets the association between the fed database object and the model (and the key in db_dict). It is quite simple.

• If is_static is True, just set _database to the fed database object.
• Otherwise, set the fed object as a proper model_id:database_object item under the proper key in app.db_dict.