4.4. Software Testing of Machine Learning Projects#
Machine learning code is very hard to test.
Due to the nature of the our models, we often have soft failures in the model that are difficult to test against. That basically means, they look like they’re doing what they’re supposed to, but secretly they’re not because of some bug.
Writing software tests in science, is already incredibly hard, so in this section we’ll touch on
some fairly simple tests we can implement to ensure consistency of our input data
avoid bad bugs in data loading procedures
some strategies to probe our models
First we’ll split the data from the Data notebook and load the model from the Sharing notebook.
from pathlib import Path
DATA_FOLDER = Path("..", "..") / "data"
DATA_FILEPATH = DATA_FOLDER / "penguins_clean.csv"
import pandas as pd
penguins = pd.read_csv(DATA_FILEPATH)
penguins.head()
| Culmen Length (mm) | Culmen Depth (mm) | Flipper Length (mm) | Sex | Species | |
|---|---|---|---|---|---|
| 0 | 39.1 | 18.7 | 181.0 | MALE | Adelie Penguin (Pygoscelis adeliae) |
| 1 | 39.5 | 17.4 | 186.0 | FEMALE | Adelie Penguin (Pygoscelis adeliae) |
| 2 | 40.3 | 18.0 | 195.0 | FEMALE | Adelie Penguin (Pygoscelis adeliae) |
| 3 | 36.7 | 19.3 | 193.0 | FEMALE | Adelie Penguin (Pygoscelis adeliae) |
| 4 | 39.3 | 20.6 | 190.0 | MALE | Adelie Penguin (Pygoscelis adeliae) |
from sklearn.model_selection import train_test_split
num_features = ["Culmen Length (mm)", "Culmen Depth (mm)", "Flipper Length (mm)"]
cat_features = ["Sex"]
features = num_features + cat_features
target = ["Species"]
X_train, X_test, y_train, y_test = train_test_split(penguins[features], penguins[target], stratify=penguins[target[0]], train_size=.7, random_state=42)
from sklearn.svm import SVC
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
import joblib
from joblib import load
MODEL_FOLDER = Path("..", "..") / "model"
MODEL_EXPORT_FILE = MODEL_FOLDER / "svc.joblib"
clf = load(MODEL_EXPORT_FILE)
clf.score(X_test, y_test)
1.0
4.4.1. Deterministic Tests#
When I work with neural networks, implementing a new layer, method, or fancy thing, I try to write a test for that layer. The Conv2D layer in Keras and Pytorch for example should always do the same exact thing, when they convole a kernel with an image.
Consider writing a small pytest test that takes a simple numpy array and tests against a known output.
You can check out the keras test suite here and an example how they validate the input and output shapes.
Admittedly this isn’t always easy to do and can go beyond the need for research scripts.
4.4.2. Data Tests for Models#
An even easier test is by essentially reusing the notebook from the Model Evaluation and writing a test function for it.
def test_penguins(clf):
# Define data you definitely know the answer to
test_data = pd.DataFrame([[34.6, 21.1, 198.0, "MALE"],
[46.1, 18.2, 178.0, "FEMALE"],
[52.5, 15.6, 221.0, "MALE"]],
columns=["Culmen Length (mm)", "Culmen Depth (mm)", "Flipper Length (mm)", "Sex"])
# Define target to the data
test_target = ['Adelie Penguin (Pygoscelis adeliae)',
'Chinstrap penguin (Pygoscelis antarctica)',
'Gentoo penguin (Pygoscelis papua)']
# Assert the model should get these right.
assert clf.score(test_data, test_target) == 1
test_penguins(clf)
This means we have some samples in the data, where we clearly know they should be part of one class and we can use these to test the model.
4.4.3. Automated Testing of Docstring Examples#
There is an even easier way to run simple tests. This can be useful when we write specific functions to pre-process our data. In the Model Sharing notebook, we looked into auto-generating docstrings.
We can upgrade our docstring and get free software tests out of it!
This is called doctest and usually useful to keep docstring examples up to date and write quick unit tests for a function.
This makes future users (including yourself from the future) quite happy.
def shorten_class_name(df: pd.DataFrame) -> pd.DataFrame:
"""Shorten the class names of the penguins to the shortest version
Parameters
----------
df : pd.DataFrame
Dataframe containing the Species column with penguins
Returns
-------
pd.DataFrame
Normalised dataframe with shortened names
Examples
--------
>>> shorten_class_name(pd.DataFrame([[1,2,3,"Adelie Penguin (Pygoscelis adeliae)"]], columns=["1","2","3","Species"]))
1 2 3 Species
0 1 2 3 Adelie
"""
df["Species"] = df.Species.str.split(r" [Pp]enguin", n=1, expand=True)[0]
return df
import doctest
doctest.testmod()
TestResults(failed=0, attempted=1)
shorten_class_name(penguins).head()
| Culmen Length (mm) | Culmen Depth (mm) | Flipper Length (mm) | Sex | Species | |
|---|---|---|---|---|---|
| 0 | 39.1 | 18.7 | 181.0 | MALE | Adelie |
| 1 | 39.5 | 17.4 | 186.0 | FEMALE | Adelie |
| 2 | 40.3 | 18.0 | 195.0 | FEMALE | Adelie |
| 3 | 36.7 | 19.3 | 193.0 | FEMALE | Adelie |
| 4 | 39.3 | 20.6 | 190.0 | MALE | Adelie |
So these give a nice example of usage in the docstring, an expected output and a first test case that is validated by our test suite.
4.4.4. Input Data Validation#
You validate that the data that users are providing matches what your model is expecting.
These tools are often used in production systems to determine whether APIs usage and user inputs are formatted correctly.
Example tools are:
import pandera as pa
# data to validate
X_train.describe()
| Culmen Length (mm) | Culmen Depth (mm) | Flipper Length (mm) | |
|---|---|---|---|
| count | 233.000000 | 233.000000 | 233.000000 |
| mean | 43.982403 | 17.228755 | 201.412017 |
| std | 5.537146 | 1.994191 | 13.929695 |
| min | 33.500000 | 13.100000 | 172.000000 |
| 25% | 39.000000 | 15.700000 | 190.000000 |
| 50% | 44.400000 | 17.300000 | 198.000000 |
| 75% | 48.800000 | 18.800000 | 213.000000 |
| max | 59.600000 | 21.200000 | 231.000000 |
The following code is supposed to fail to see what happens if the schema doesn’t match!
# define schema
schema = pa.DataFrameSchema({
"Culmen Length (mm)": pa.Column(float, checks=[pa.Check.ge(30),
pa.Check.le(60)]),
"Culmen Depth (mm)": pa.Column(float, checks=[pa.Check.ge(13),
pa.Check.le(22)]),
"Flipper Length (mm)": pa.Column(float, checks=[pa.Check.ge(170),
pa.Check.le(235)]),
"Sex": pa.Column(str, checks=pa.Check.isin(["MALE","FEMALE"])),
})
validated_test = schema(X_test)
---------------------------------------------------------------------------
SchemaError Traceback (most recent call last)
Cell In[11], line 12
1 # define schema
2 schema = pa.DataFrameSchema({
3 "Culmen Length (mm)": pa.Column(float, checks=[pa.Check.ge(30),
4 pa.Check.le(60)]),
(...)
9 "Sex": pa.Column(str, checks=pa.Check.isin(["MALE","FEMALE"])),
10 })
---> 12 validated_test = schema(X_test)
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/api/pandas/container.py:443, in DataFrameSchema.__call__(self, dataframe, head, tail, sample, random_state, lazy, inplace)
415 def __call__(
416 self,
417 dataframe: pd.DataFrame,
(...)
423 inplace: bool = False,
424 ):
425 """Alias for :func:`DataFrameSchema.validate` method.
426
427 :param pd.DataFrame dataframe: the dataframe to be validated.
(...)
441 otherwise creates a copy of the data.
442 """
--> 443 return self.validate(
444 dataframe, head, tail, sample, random_state, lazy, inplace
445 )
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/api/pandas/container.py:375, in DataFrameSchema.validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
363 check_obj = check_obj.map_partitions( # type: ignore [operator]
364 self._validate,
365 head=head,
(...)
371 meta=check_obj,
372 )
373 return check_obj.pandera.add_schema(self)
--> 375 return self._validate(
376 check_obj=check_obj,
377 head=head,
378 tail=tail,
379 sample=sample,
380 random_state=random_state,
381 lazy=lazy,
382 inplace=inplace,
383 )
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/api/pandas/container.py:404, in DataFrameSchema._validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
395 if self._is_inferred:
396 warnings.warn(
397 f"This {type(self)} is an inferred schema that hasn't been "
398 "modified. It's recommended that you refine the schema "
(...)
401 UserWarning,
402 )
--> 404 return self.get_backend(check_obj).validate(
405 check_obj,
406 schema=self,
407 head=head,
408 tail=tail,
409 sample=sample,
410 random_state=random_state,
411 lazy=lazy,
412 inplace=inplace,
413 )
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/backends/pandas/container.py:97, in DataFrameSchemaBackend.validate(self, check_obj, schema, head, tail, sample, random_state, lazy, inplace)
92 components = self.collect_schema_components(
93 check_obj, schema, column_info
94 )
96 # run the checks
---> 97 error_handler = self.run_checks_and_handle_errors(
98 error_handler,
99 schema,
100 check_obj,
101 column_info,
102 sample,
103 components,
104 lazy,
105 head,
106 tail,
107 random_state,
108 )
110 if error_handler.collected_errors:
111 if getattr(schema, "drop_invalid_rows", False):
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/backends/pandas/container.py:172, in DataFrameSchemaBackend.run_checks_and_handle_errors(self, error_handler, schema, check_obj, column_info, sample, components, lazy, head, tail, random_state)
161 else:
162 error = SchemaError(
163 schema,
164 data=check_obj,
(...)
170 reason_code=result.reason_code,
171 )
--> 172 error_handler.collect_error(
173 result.reason_code,
174 error,
175 original_exc=result.original_exc,
176 )
178 return error_handler
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/error_handlers.py:38, in SchemaErrorHandler.collect_error(self, reason_code, schema_error, original_exc)
31 """Collect schema error, raising exception if lazy is False.
32
33 :param reason_code: string representing reason for error.
34 :param schema_error: ``SchemaError`` object.
35 :param original_exc: original exception associated with the SchemaError.
36 """
37 if not self._lazy:
---> 38 raise schema_error from original_exc
40 # delete data of validated object from SchemaError object to prevent
41 # storing copies of the validated DataFrame/Series for every
42 # SchemaError collected.
43 del schema_error.data
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/backends/pandas/container.py:192, in DataFrameSchemaBackend.run_schema_component_checks(self, check_obj, schema_components, lazy)
190 for schema_component in schema_components:
191 try:
--> 192 result = schema_component.validate(
193 check_obj, lazy=lazy, inplace=True
194 )
195 check_passed.append(is_table(result))
196 except SchemaError as err:
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/api/pandas/components.py:169, in Column.validate(self, check_obj, head, tail, sample, random_state, lazy, inplace)
142 def validate(
143 self,
144 check_obj: pd.DataFrame,
(...)
150 inplace: bool = False,
151 ) -> pd.DataFrame:
152 """Validate a Column in a DataFrame object.
153
154 :param check_obj: pandas DataFrame to validate.
(...)
167 :returns: validated DataFrame.
168 """
--> 169 return self.get_backend(check_obj).validate(
170 check_obj,
171 self,
172 head=head,
173 tail=tail,
174 sample=sample,
175 random_state=random_state,
176 lazy=lazy,
177 inplace=inplace,
178 )
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/backends/pandas/components.py:119, in ColumnBackend.validate(self, check_obj, schema, head, tail, sample, random_state, lazy, inplace)
115 check_obj = validate_column(
116 check_obj, column_name, return_check_obj=True
117 )
118 else:
--> 119 validate_column(check_obj, column_name)
121 if lazy and error_handler.collected_errors:
122 raise SchemaErrors(
123 schema=schema,
124 schema_errors=error_handler.collected_errors,
125 data=check_obj,
126 )
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/backends/pandas/components.py:89, in ColumnBackend.validate.<locals>.validate_column(check_obj, column_name, return_check_obj)
84 error_handler.collect_error(
85 reason_code=None,
86 schema_error=err,
87 )
88 except SchemaError as err:
---> 89 error_handler.collect_error(err.reason_code, err)
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/error_handlers.py:38, in SchemaErrorHandler.collect_error(self, reason_code, schema_error, original_exc)
31 """Collect schema error, raising exception if lazy is False.
32
33 :param reason_code: string representing reason for error.
34 :param schema_error: ``SchemaError`` object.
35 :param original_exc: original exception associated with the SchemaError.
36 """
37 if not self._lazy:
---> 38 raise schema_error from original_exc
40 # delete data of validated object from SchemaError object to prevent
41 # storing copies of the validated DataFrame/Series for every
42 # SchemaError collected.
43 del schema_error.data
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/backends/pandas/components.py:68, in ColumnBackend.validate.<locals>.validate_column(check_obj, column_name, return_check_obj)
65 def validate_column(check_obj, column_name, return_check_obj=False):
66 try:
67 # pylint: disable=super-with-arguments
---> 68 validated_check_obj = super(ColumnBackend, self).validate(
69 check_obj,
70 copy(schema).set_name(column_name),
71 head=head,
72 tail=tail,
73 sample=sample,
74 random_state=random_state,
75 lazy=lazy,
76 inplace=inplace,
77 )
79 if return_check_obj:
80 return validated_check_obj
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/backends/pandas/array.py:69, in ArraySchemaBackend.validate(self, check_obj, schema, head, tail, sample, random_state, lazy, inplace)
66 error_handler.collect_error(exc.reason_code, exc)
68 # run the core checks
---> 69 error_handler = self.run_checks_and_handle_errors(
70 error_handler,
71 schema,
72 check_obj,
73 head,
74 tail,
75 sample,
76 random_state,
77 )
79 if lazy and error_handler.collected_errors:
80 if getattr(schema, "drop_invalid_rows", False):
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/backends/pandas/array.py:150, in ArraySchemaBackend.run_checks_and_handle_errors(self, error_handler, schema, check_obj, head, tail, sample, random_state)
139 else:
140 error = SchemaError(
141 schema=schema,
142 data=check_obj,
(...)
148 reason_code=result.reason_code,
149 )
--> 150 error_handler.collect_error(
151 result.reason_code,
152 error,
153 original_exc=result.original_exc,
154 )
156 return error_handler
File /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pandera/error_handlers.py:38, in SchemaErrorHandler.collect_error(self, reason_code, schema_error, original_exc)
31 """Collect schema error, raising exception if lazy is False.
32
33 :param reason_code: string representing reason for error.
34 :param schema_error: ``SchemaError`` object.
35 :param original_exc: original exception associated with the SchemaError.
36 """
37 if not self._lazy:
---> 38 raise schema_error from original_exc
40 # delete data of validated object from SchemaError object to prevent
41 # storing copies of the validated DataFrame/Series for every
42 # SchemaError collected.
43 del schema_error.data
SchemaError: <Schema Column(name=Sex, type=DataType(str))> failed element-wise validator 0:
<Check isin: isin(['MALE', 'FEMALE'])>
failure cases:
index failure_case
0 259 .
X_test.Sex.unique()
array(['FEMALE', 'MALE', '.'], dtype=object)
X_test.loc[259]
Culmen Length (mm) 44.5
Culmen Depth (mm) 15.7
Flipper Length (mm) 217.0
Sex .
Name: 259, dtype: object
Can you fix the data to conform to the schema?