What's your data preprocessing strategy?

Data preprocessing is a crucial step in any machine learning project. It involves transforming, cleaning, and organizing your raw data into a suitable format for your algorithms. Without proper data preprocessing, you risk compromising the quality and accuracy of your results, wasting time and resources, and missing valuable insights. In this article, we will discuss some common data preprocessing techniques and how to choose the best strategy for your data and goals.

1 Data types

The first thing you need to do is to identify the types of data you have and how they relate to your problem. Data types can be broadly classified into numerical, categorical, or text data. Numerical data can be further divided into continuous or discrete, depending on whether they have a finite or infinite range of values. Categorical data can be either nominal or ordinal, depending on whether they have a natural order or not. Text data can be either structured or unstructured, depending on whether they follow a predefined format or not. Knowing your data types will help you decide what kind of preprocessing techniques you need to apply, such as scaling, encoding, or tokenizing.

Add your perspective

2 Data quality

The next thing you need to do is to assess the quality of your data and fix any issues that might affect your analysis. Data quality issues can include missing values, outliers, duplicates, inconsistencies, errors, or noise. Depending on the nature and extent of these issues, you can use different methods to deal with them, such as imputing, removing, correcting, or smoothing. Data quality issues can have a significant impact on your machine learning performance, so you need to be careful and thorough when handling them.

Add your perspective

3 Data exploration

The third thing you need to do is to explore your data and understand its characteristics and patterns. Data exploration involves using descriptive statistics, visualizations, and correlations to summarize and display your data. Data exploration can help you gain insights into your data, such as its distribution, variability, trends, relationships, or anomalies. Data exploration can also help you identify potential features, targets, or hypotheses for your machine learning models.

Add your perspective

Lalith Sharan

Doctoral Student at Heidelberg University
Report contribution
Exploratory data analysis is a mandatory step before trying to force feed your model with unclean or incomplete data. Here, it helps to firstly understand the distribution of the dataset, by plotting histograms for example, and looking for any obvious outliers. Secondly, visualising the features and their occurrences also helps understand the data better. Here, the seaborn or matplotlib library of python offers functions like pair plots. At every stage you need to ask yourself if the data distribution violates any assumptions about your underlying model of the world. Finally, dimensionality reduction methods like PCA or t-SNE visualisations help you not only gain insights from data but also choose the potentially important features.

Like

Unhelpful

4 Data transformation

The fourth thing you need to do is to transform your data into a format that is suitable for your machine learning algorithms. Data transformation involves applying mathematical or logical operations to your data, such as scaling, normalizing, standardizing, binning, or log-transforming. Data transformation can help you improve the performance and stability of your machine learning models, by reducing the effect of outliers, skewness, or different scales.

Add your perspective

5 Data encoding

The fifth thing you need to do is to encode your data into a numerical representation that your machine learning algorithms can understand. Data encoding involves converting your categorical or text data into numeric values, such as binary, one-hot, ordinal, or frequency encoding. Data encoding can help you capture the information and meaning of your data, by preserving the order, similarity, or importance of the categories or words.

Add your perspective

6 Data splitting

The last thing you need to do is to split your data into different subsets for training, validation, and testing your machine learning models. Data splitting involves dividing your data into random and representative samples, using a predefined ratio or method, such as holdout, k-fold, or stratified. Data splitting can help you evaluate and compare the performance and generalization of your machine learning models, by avoiding overfitting, underfitting, or bias.

Add your perspective

Lalith Sharan

Doctoral Student at Heidelberg University
Report contribution
Data splitting is not always trivial. In case of small datasets, cross-validation may work better, whereas for large datasets, you would rather have a completely unseen test dataset to evaluate the generalisation ability of the model. Firstly, you need to ensure that there is no data leakage. This may sound obvious but let us take an example of image data in the medical domain. A random splitting or a shuffle and split may still mean that more than one image belonging to the same patient or the same surgery may fall into different splits, thereby corrupting your evaluation. One needs to avoid these cases by testing for data leakage.

Like

Unhelpful

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

Add your perspective

Lalith Sharan

Doctoral Student at Heidelberg University
Report contribution
Often there is a data shift between the train and test sets because they come from slightly different distributions. Here, it needs to be ensured that the different splits reflect the distribution of different classes of the datasets. This can be approached by stratifying the splitting. The sklearn library of python offers nice functions for this purpose, for example sklearn.model_selection.StratifiedShuffleSplit. Sometimes stratification needs to be performed on multiple levels to ensure hierarchical relations are also maintained and reflected in your splits.

Like

Unhelpful

What's your data preprocessing strategy?

1

2

3

4

5

6

7

1 Data types

2 Data quality

3 Data exploration

4 Data transformation

5 Data encoding

6 Data splitting

7 Here’s what else to consider

Machine Learning

Rate this article

Thanks for your feedback

More articles on Machine Learning

More relevant reading

What's your data preprocessing strategy?

1

2

3

4

5

6

7

1 Data types

2 Data quality

3 Data exploration

4 Data transformation

5 Data encoding

6 Data splitting

7 Here’s what else to consider

Machine Learning

Rate this article

Thanks for your feedback

Explore Other Skills