Home / MAD1 / Week 10

Week 10 · Testing web applications with pytest

The one-minute story

By the end of this week you can:

• explain why we test and name the kinds of tests (unit, integration, system, regression, acceptance);
• write pytest tests with assert, fixtures, parametrize and conftest.py;
• test Flask pages and JSON APIs with app.test_client() against a separate in-memory SQLite database;
• predict the exact summary line of pytest -k ... and pytest -m ... runs;
• explain mocking, test doubles, statement vs branch coverage, and continuous testing.

Words you need

Idea 1: Why test, and the kinds of tests

Without tests you find bugs when a user complains. With tests you find them in seconds, on your own laptop. Tests also let you change code without fear: if all tests still pass, you probably broke nothing.

Testing pyramid. Write many unit tests (cheap, fast), fewer integration tests, and very few end-to-end tests (slow, fragile). The picture is a pyramid with unit tests as the wide base.

White box vs black box. If you design tests by reading the code (every if branch must run), it is white box testing. If you only know "this input should give that output", it is black box testing.

Test-driven development (TDD) is a habit with three steps, repeated in short loops:

1. Red: write a test for a feature that does not exist yet. Run it. It fails.
2. Green: write the smallest code that makes the test pass.
3. Refactor: clean the code while keeping all tests green.

Continuous testing. Tests are run automatically on every push (see CI in week 12). Continuous integration means changes from many people are merged into one code base often and tested each time. Continuous delivery means that after the tests pass, a release package is prepared automatically. Continuous deployment goes one step further: every passing change is put into production automatically.

Idea 2: pytest basics

Install: pip install pytest. pytest is the tool; the tests are plain functions with plain assert.

# file: test_cases.py
def square(n):
    return n * n

def test1():
    assert square(5) == 25

def test2():
    assert square(square(2)) == 16

def test3():
    assert square(square(2)) == 8

Run pytest -v test_cases.py. Real output (trimmed):

test_cases.py::test1 PASSED                                              [ 33%]
test_cases.py::test2 PASSED                                              [ 66%]
test_cases.py::test3 FAILED                                              [100%]
FAILED test_cases.py::test3 - assert 16 == 8
========================= 1 failed, 2 passed in 0.09s =========================

square(square(2)) is square(4) which is 16, so test3 fails and pytest shows both sides of the broken comparison: assert 16 == 8.

What the summary line means. Common words: passed, failed, deselected (collected but skipped because of -k or -m), error (a problem outside the test body, such as a missing fixture), warnings.

# test_class_demo.py
class TestGroup:
    def do_something(self, x, y):      # helper, not a test (no test_ prefix)
        return x % y

    def test_one(self):
        assert self.do_something(10, 5) == 0, "Is 5 factor of 10?"

    def test_two(self):
        assert self.do_something(32, 5) == 0, "Is 5 factor of 32?"

pytest -v prints TestGroup::test_one PASSED and TestGroup::test_two FAILED; the failure line carries your message after AssertionError:. Only the failed test's message is shown; the passing test's message never appears. Summary: 1 failed, 1 passed.

Useful pytest pieces

import pytest

def test_float():
    assert 0.1 + 0.2 == pytest.approx(0.3)      # never compare floats with ==

def test_error():
    with pytest.raises(ZeroDivisionError):      # passes only if the error is raised
        1 / 0

def test_tmp(tmp_path):                         # built-in fixture: a fresh temp folder
    f = tmp_path / "notes.txt"
    f.write_text("hi")
    assert f.read_text() == "hi"

tmp_path gives every test its own empty temporary folder that is cleaned up later, so tests that write files never disturb each other or your real files. pytest also has monkeypatch (temporarily replace a variable or function) and capsys (capture print output).

def dec(x):
    return x - 1

def test_answer():
    assert dec(3) == 3

Idea 3: Fixtures, conftest.py, parametrize, markers, -k and -m

A fixture prepares something and hands it to the test. A test asks for it by writing the fixture's name as a parameter.

import pytest

@pytest.fixture
def students():
    return {"names": ["ravi", "raj"], "ages": [12, 9]}

def test_name(students):
    assert "ravi" in students["names"]

def test_age(students):
    assert students["ages"][0] > 9

Each test gets a fresh copy (the default scope is per test), so one test cannot damage another's data. Fixtures that need cleanup use yield: the code before yield is the setup, the value after yield is given to the test, and the code after yield runs as teardown.

@pytest.fixture
def numbers():
    print("setup")
    yield [1, 2, 3]
    print("teardown")     # runs after the test finishes

Parametrize runs one test with many inputs:

@pytest.mark.parametrize("a,b,expected", [(2, 3, 5), (0, 0, 0), (-1, 1, 0)])
def test_add_numbers(a, b, expected):
    assert a + b == expected

This is three tests, shown as test_add_numbers[2-3-5], test_add_numbers[0-0-0], test_add_numbers[-1-1-0]. Real run: all pass.

Markers label tests: @pytest.mark.slow. Select with -m. Register your markers in pytest.ini to avoid warnings:

[pytest]
markers =
    slow: tests that take long
    unit: unit tests

Selecting tests (this is what the papers ask the most):

import pytest

def square(x):
    sum = 0
    for counter in range(x):
        sum += x
    return sum        # this really computes x times x

@pytest.mark.marker1
def testcase_1(): assert square(10) == 100
@pytest.mark.marker2
def testcase_2(): assert square(4) == 4
@pytest.mark.marker3
def testcase_3(): assert square(5) == 25

def func(x, y):
    return x**2 + y**2

class Test_class0:
    def test_case1(self): assert func(1, 2) == 5      # 1+4 = 5, pass
    def case_test2(self): assert func(2, 3) == 13     # not collected
    def test_case3(self): assert func(6, 2) == 38     # 36+4 = 40, fail

class Test_class1:
    def test_case1(self): assert func(5, 2) == 29     # 25+4 = 29, pass
    def case_test2(self): assert func(1, 1) == 2      # not collected

Idea 4: Testing Flask with the test client

app.test_client() returns an object that behaves like a browser but calls your app directly in memory. No server, no port, very fast.

Here is the small app we test (an application factory, so a test can build its own app):

# app.py
from flask import Flask, jsonify, request, redirect, url_for
from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()

class Todo(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(80), nullable=False)

def create_app(config=None):
    app = Flask(__name__)
    app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///todo.db"
    if config:
        app.config.update(config)           # a test can override the database
    db.init_app(app)
    with app.app_context():
        db.create_all()

    @app.route("/")
    def home():
        return "Todo app"

    @app.route("/add", methods=["POST"])
    def add():
        title = request.form.get("title", "").strip()
        if not title:
            return "Title required", 400
        db.session.add(Todo(title=title))
        db.session.commit()
        return redirect(url_for("home"))

    @app.route("/api/todos", methods=["GET", "POST"])
    def todos():
        if request.method == "POST":
            data = request.get_json()
            if not data or "title" not in data:
                return jsonify(error="title missing"), 400
            t = Todo(title=data["title"])
            db.session.add(t)
            db.session.commit()
            return jsonify(id=t.id, title=t.title), 201
        return jsonify([{"id": t.id, "title": t.title} for t in Todo.query.all()])

    return app

The key trick: a separate in-memory database for tests. If tests used todo.db, they would fill your real data with junk. The URI sqlite:///:memory: creates a database that lives only in RAM and vanishes when the app object goes away. Every test gets a fresh, empty database. (Flask-SQLAlchemy 3 keeps one shared connection for in-memory SQLite, so the tables you create stay visible.)

# conftest.py
import pytest
from app import create_app, db

@pytest.fixture
def app():
    app = create_app({"TESTING": True, "SQLALCHEMY_DATABASE_URI": "sqlite:///:memory:"})
    yield app
    with app.app_context():
        db.drop_all()

@pytest.fixture
def client(app):
    return app.test_client()

TESTING=True makes Flask let exceptions propagate to the test instead of hiding them behind a 500 page. A fixture can use another fixture (client asks for app).

# test_app.py
def test_home(client):
    resp = client.get("/")
    assert resp.status_code == 200
    assert b"Todo" in resp.data           # resp.data is BYTES

def test_add_requires_title(client):
    resp = client.post("/add", data={"title": "  "})
    assert resp.status_code == 400

def test_add_redirects(client):
    resp = client.post("/add", data={"title": "milk"})
    assert resp.status_code == 302
    assert resp.headers["Location"] == "/"

def test_add_follow(client):
    resp = client.post("/add", data={"title": "milk"}, follow_redirects=True)
    assert resp.status_code == 200
    assert resp.data == b"Todo app"

All of these passed in a real run. Notes:

• client.get(url) and client.post(url, data={...}) return a response. data={...} sends a form (like an HTML form post).
• resp.status_code is the HTTP status (200, 302, 400, 404, 405...). resp.data is the body as bytes (b"..."); resp.get_data(as_text=True) gives text. resp.headers holds headers.
• A redirect is not followed by default: you see the 302 and the Location header. With follow_redirects=True the client follows it and you see the final page.
• Posting to a route that allows only GET gives 405; an unknown URL gives 404.

Testing JSON API endpoints

def test_api_create_and_list(client):
    r = client.post("/api/todos", json={"title": "milk"})   # json= sends a JSON body
    assert r.status_code == 201
    assert r.get_json() == {"id": 1, "title": "milk"}
    r = client.get("/api/todos")
    assert r.get_json() == [{"id": 1, "title": "milk"}]

def test_api_bad(client):
    r = client.post("/api/todos", json={})
    assert r.status_code == 400
    assert r.get_json()["error"] == "title missing"

def test_isolated(client):
    assert client.get("/api/todos").get_json() == []         # fresh database again

json= sets the Content-Type: application/json header and encodes the body; r.get_json() decodes the reply. The id is 1 in the first test, and test_isolated sees an empty list, proving each test has its own database. Whole file: 10 passed.

Idea 5: Mocking and test doubles

A good unit test must be fast, repeatable and independent of the outside world. Calling a real weather service, sending a real e-mail or reading the real clock breaks all three. So we replace those parts with fakes. All the fakes are called test doubles:

# weather.py
import requests

def temperature(city):
    r = requests.get("https://api.example.com/temp", params={"city": city}, timeout=5)
    r.raise_for_status()
    return r.json()["celsius"]

def describe(city):
    t = temperature(city)
    return "hot" if t >= 30 else "ok"

# test_weather.py
from unittest.mock import patch, MagicMock
import weather

@patch("weather.requests.get")                    # replace requests.get inside weather.py
def test_describe_hot(mock_get):
    mock_get.return_value = MagicMock(status_code=200)
    mock_get.return_value.json.return_value = {"celsius": 35}
    assert weather.describe("Chennai") == "hot"
    mock_get.assert_called_once()

def test_describe_ok(monkeypatch):
    monkeypatch.setattr(weather, "temperature", lambda city: 20)
    assert weather.describe("Ooty") == "ok"

Both pass in a real run, with no internet used. The golden rule: patch where the name is used (weather.requests.get), not where it was defined. Mocks are also the way to force rare cases (a timeout, a 500 from a service).

Idea 6: Code coverage, UI tests, and the habit of running tests

Code coverage answers "how much of my code did the tests execute?". With the pytest-cov plugin:

pip install pytest-cov
pytest --cov=app --cov-branch --cov-report=term-missing

Real output on our app with the 10 tests above:

Name     Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------
app.py      38      0      8      1    98%   14->16
----------------------------------------------------
TOTAL       38      0      8      1    98%

All 38 statements ran (Miss 0), but one of 8 branches was only taken one way: line 14 is if config:, and our tests always pass a config, so the "no config" path was never taken (14->16 means "from line 14 jump to line 16" was never seen). That is why statement coverage is 100 percent yet total coverage shows 98 percent once branches count.

UI testing with Selenium (concept). Selenium lets a script start a real browser, open your site, fill the form, click buttons and read the page. It tests what a user sees (HTML, CSS, JavaScript together). It is slow and fragile (a changed button label breaks it), so you write few of them. In the syllabus it is "system testing automation".

# concept only; needs a browser driver installed
from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("http://127.0.0.1:5000/login")
driver.find_element(By.NAME, "email").send_keys("a@b.com")
driver.find_element(By.ID, "login-btn").click()
assert "Dashboard" in driver.title
driver.quit()

Your turn

def test_user_add(): assert True
def test_user_del(): assert False
def test_item_add(): assert True

def test_missing_page(client):
    assert client.get("/nothing").status_code == 404

powers = []
for i in range(1, 5):
    def powers_of_x(x):
        return x ** i
    powers.append(powers_of_x)
powers_of_3 = [f(3) for f in powers]

Exam radar

Quick quiz

Viva: say it out loud

Recap

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 60 real questions on this week   All End Term papers