Understanding Temporal Dependencies in Longitudinal Datasets¶

In the following, we will look at the only component of Sklong that is completely novel compared to previous libraries. Our explanation covers how we generalise the use of temporal dependencies in longitudinal datasets across all Sklong algorithms, methods, and primitives.

This reading aims to help you establish up longitudinal datasets in 'Sklong' that allow you to utilise any algorithm without worrying about temporal dependencies. The primitives will handle this based on their design.

Understanding features_group¶

Before the formal definition, here is the same idea in one pass: a wide-format table, with each longitudinal attribute's columns collapsed into one inner list of indices (oldest → most recent), and static covariates kept aside in non_longitudinal_features.

features_group is a list of lists of integers, with each inner list representing a group of features for a specific longitudinal variable. The inner lists' indices are ordered by wave/time-point sequence, capturing the temporal dependencies required for longitudinal data algorithms.

Consider a dataset with four features, two of which are longitudinal and each consist of two records collected over time, called waves or time-points. A real-world example would be smoke and cholesterol, each with two waves/time-points, and you want to divide them into two groups, one with the first longitudinal attribute, smoke, which is made up of the two feature indices in the dataset about smoke, and the other with the second l ongitudinal attribute, cholesterol. In this case, you would pass the following list of lists of integers as the features_group parameter:

[[0,1],[2,3]]

Here, 0 and 1 are the indices of the first longitudinal attribute smoke, and 2 and 3 are the indices of the second longitudinal attribute cholesterol. So 0 is smoke wave/time-point 1, 1 is smoke wave/time-point 2, 2 is cholesterol wave/time-point 1, and 3 is cholesterol wave/time-point 2. Hence, the algorithm can deal with the feature recentness, i.e., the first element of the inner lists are older, and the farther the element is from the first element, the more recent it is.

Understanding non_longitudinal_features¶

non_longitudinal_features contains indices for non-temporal features. These features have no temporal order and can be handled separately by the algorithms. However, how these features are treated is determined by the algorithm designer. This means that algorithm designers can or cannot incorporate these features into their algorithms, watch out for the algorithm's documentation to know how it handles parameters.

To come back to the object. For example, if you have a dataset with 5 features, where the first 4 are longitudinal attributes (features_group as [[0,1],[2,3]]), and the last one is non-longitudinal, you would pass the following list of integers in the non_longitudinal_features parameter:

[4]

Let's take an exemplary dataset¶

Now let's set up the features_group and non_longitudinal_features for this dataset for Sklong:

from scikit_longitudinal.data_preparation import LongitudinalDataset

dataset = LongitudinalDataset('./stroke_longitudinal.csv')
dataset.load_data()

# Manually set your temporal dependencies
dataset.setup_features_group(
 input_data=[[2,3],[4,5],[6,7],[8,9],[10,11],[12,13]],
)

print(f"Features group: {dataset.feature_groups(names=True)}")
>$ Features group: [['smoke_w1', 'smoke_w2'], ['cholesterol_w1', 'cholesterol_w2'], ['blood_pressure_w1', 'blood_pressure_w2'], ['diabetes_w1', 'diabetes_w2'], ['exercise_w1', 'exercise_w2'], ['obesity_w1', 'obesity_w2']]
print(f"Non-longitudinal features: {dataset.non_longitudinal_features(names=True)}")
>$ Non-longitudinal features: ['age', 'gender']

Pre-set features_group and non_longitudinal_features¶

Presets are configurations that can be used to quickly set up the features_group and non_longitudinal_features based on a given popular database.

We currently have a pre-set configuration for the features_group and non_longitudinal_features in the English Longitudinal Study of Ageing (ELSA) database. The ELSA database is an ageing-related diseases longitudinal database that can be accessed via this link: ELSA.

The ELSA database tracks core participants, who are 50 years of age or older and reside in the United Kingdom, through repeated interviews. For instance, biomedical data collected every four years by a nurse or health professional results in ELSA-nurse datasets, while data from core interviews conducted every two years results in ELSA-core datasets.

Instead of using your own configuration for the input_data parameter of the setup_features_group method of LongitudinalDataset, you can use the pre-set configuration for the ELSA database, which is passed as a string to the input_data parameter. It will generate the features_group and non_longitudinal_features for you based on how the data is constructed. An exemplary usage is shown below:

from scikit_longitudinal.data_preparation import LongitudinalDataset

dataset = LongitudinalDataset('./extended_stroke_longitudinal.csv')
dataset.load_data()

# Pre-set your temporal dependencies
dataset.setup_features_group(input_data="elsa")

print(f"Features group: {dataset.feature_groups(names=True)}")
>$ ...will print the features group of the ELSA dataset ...
print(f"Non-longitudinal features: {dataset.non_longitudinal_features(names=True)}")
>$ ...will print the non-longitudinal features of the ELSA dataset ...

To conclude, the appropriate configuration of features_group and non_longitudinal_features is critical for algorithm designers and library users. It allows anyone to use the temporal structure of longitudinal data as well as non-temporal features to capture the underlying patterns in the data using any adapted/newly designed algorithms for longitudinal data classification tasks in a shared-common and user-friendly manner. As a result, instead of having to create an algorithm that only works on one dataset by, for example, hard-coding the dataset's temporal structure, another one could, for example, rely on the feature names, rendering it inapplicable to other datasets.