Home / MAD2 / Week 10

Week 10 · Messaging: webhooks, server-sent events and email

The one-minute story

By the end of this week you can:

• explain synchronous vs asynchronous messaging, and queue vs publish/subscribe;
• write a webhook sender (Python requests) and a webhook receiver (Flask) that checks a signature;
• write a Flask SSE endpoint and the browser EventSource that reads it, and choose between SSE, WebSockets, polling and long polling;
• send an HTML email from Flask and test it with MailHog;
• schedule a daily reminder and a monthly report with Celery beat, and export a PDF or CSV;
• predict the answer to the typical "which statement about webhooks / SSE is true" question.

Words you need

Idea 1: Services talking to each other

A real application is rarely one program. In your project there is a Flask API, a Celery worker, Redis, maybe a mail server. They need to pass messages. There are two styles.

Queue vs publish/subscribe

Brokers you may hear about: Redis (queues, lists and pub/sub, the one used in your course), RabbitMQ (a full message broker, a common Celery broker), Kafka (a durable log for very high volume). For the course you only need to know what they are for.

Redis pub/sub in a few lines

This needs a Redis server running (see the week 12 lesson for Windows tips). The redis package is pip install redis.

# subscriber.py (run first, in terminal 1)
import redis
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
p = r.pubsub()
p.subscribe("orders")
for msg in p.listen():
    if msg["type"] == "message":
        print("got:", msg["data"])

# publisher.py (terminal 2)
import redis
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
n = r.publish("orders", "order 42 placed")
print("delivered to", n, "subscribers")

Terminal 1 prints got: order 42 placed. Terminal 2 prints delivered to 1 subscribers. If no one is subscribed, the number is 0 and the message is lost: Redis pub/sub does not store messages. (A Celery queue, in contrast, keeps the task until a worker takes it.)

Idea 2: Webhooks

Webhook vs API call vs polling

"A webhook is triggered by an event, while an API call requires a client request" is the textbook difference. A webhook is not the same as short polling, and not the same as a WebSocket.

Sending a webhook with Python requests

Google Chat and Slack both give you an incoming webhook URL. You POST a small JSON message to it and the text appears in a chat room. The URL is a secret: anyone who has it can post. Keep it in an environment variable, never in Git.

import os
import requests

URL = os.environ["CHAT_WEBHOOK_URL"]   # looks like https://chat.googleapis.com/v1/spaces/.../messages?key=...

def notify(text):
    r = requests.post(URL, json={"text": text}, timeout=5)
    r.raise_for_status()               # turns 4xx/5xx into an exception
    return r.status_code

notify("Daily report is ready: 12 users inactive.")

json= sets the body to JSON and the header Content-Type: application/json. Google Chat accepts {"text": "..."}; Slack incoming webhooks also take {"text": "..."}. Always pass a timeout, otherwise a dead server can hang your worker forever.

Receiving a webhook in Flask, with a signature check

Anyone on the internet can POST to your receiver URL. How do you know it was really the sender? The sender and you share a secret. The sender computes an HMAC of the raw body with the secret and sends it in a header. You recompute and compare.

import hmac, hashlib
from flask import Flask, request, jsonify

app = Flask(__name__)
SECRET = b"s3cret-shared-key"          # in real code: os.environ

def sign(body: bytes) -> str:
    return hmac.new(SECRET, body, hashlib.sha256).hexdigest()

@app.post("/hooks/payment")
def payment_hook():
    body = request.get_data()                        # the RAW bytes
    got = request.headers.get("X-Signature", "")
    if not hmac.compare_digest(got, sign(body)):
        return jsonify(error="bad signature"), 401
    event = request.get_json()
    print("event:", event["type"])
    return jsonify(ok=True), 200                     # answer quickly

I ran this with Flask's test client: a body signed with the right secret gives 200, a body with no signature gives 401.

Why these details:

• Sign the raw bytes (request.get_data()), not a re-dumped JSON, because spaces and key order change the hash.
• hmac.compare_digest compares in constant time, so an attacker cannot learn the signature by timing.
• Return 200 fast (within a few seconds). Do slow work in a Celery task: process_event.delay(event). Senders give up on slow receivers.
• Use HTTPS, so the secret payload is not readable on the way.

Retries and idempotency

processed = set()          # in real code: a database table or Redis set

def handle(event):
    if event["id"] in processed:
        return "duplicate, ignored"
    processed.add(event["id"])
    # ... do the work once ...
    return "done"

Calling handle({"id": "evt_1"}) twice returns done then duplicate, ignored.

Idea 3: Server-Sent Events (SSE)

Polling asks again and again. A webhook goes server to server. But the browser is not a server, it cannot receive a POST. How does a page learn "your export is done" without reloading? One answer is SSE.

The wire format

data: Order placed

data: Preparing

event: done
data: {"status": "delivered"}

Each event ends with an empty line. A line data: x fires onmessage. A line event: done gives it a name, and the browser must listen with addEventListener("done", ...). An optional id: 7 line lets the browser tell the server where it stopped (the Last-Event-ID header on reconnect). A line retry: 3000 sets the reconnect delay in milliseconds.

Flask streaming generator

import time
from flask import Flask, Response

app = Flask(__name__)

def event_stream():
    for i in range(5):
        yield f"data: Message {i}\n\n"     # note the blank line
        time.sleep(1)

@app.get("/events")
def events():
    return Response(event_stream(), mimetype="text/event-stream")

A view that returns a generator inside a Response streams each yield to the client as it is produced. I checked that the response header is text/event-stream and the body is the five data: blocks.

The browser side (plain JavaScript and Vue 2)

const es = new EventSource("http://localhost:5000/events");
es.onmessage = (event) => console.log(event.data);   // "Message 0", "Message 1", ...
es.onerror = () => console.log("lost connection, the browser will retry");
// es.close();   // stop listening

// inside a Vue 2 component
export default {
  data() { return { messages: [] }; },
  mounted() {
    this.es = new EventSource("http://localhost:5000/events");
    this.es.onmessage = (e) => { this.messages.push(e.data); };
  },
  beforeDestroy() { if (this.es) this.es.close(); }
};

event.data is always a string: send JSON text and use JSON.parse(event.data).

SSE with many clients and Redis

A generator like the one above is tied to one process. If a worker or another request must push an event to many connected browsers, they need a shared channel. The Flask-SSE extension does this: you call sse.publish({"msg": "low stock"}, type="alert") anywhere, and Redis acts as the shared channel (pub/sub) that carries the message to every web process holding open connections. Redis here is a message channel, not a login checker, and not mandatory storage of old events. Flask-SSE needs a streaming server (gunicorn with gevent), not the plain development server with many clients.

SSE vs WebSockets vs polling vs long polling

• Long polling: the client makes a single request and the server holds it open until new data exists, then answers; the client immediately asks again. (Not "repeated requests at fixed intervals": that is short polling.)
• When people just say "polling" they mean short polling, over HTTP.
• SSE is not an extension of WebSockets and does not need service workers or HTTP/2. It cannot send data from the client, but the client can use a normal fetch POST for that.
• SSE and the WebSocket handshake both start with an HTTP GET.
• A food-delivery tracker (server pushes "Preparing", "Out for delivery") is a textbook SSE case: one direction is enough.

Idea 4: Sending email from Flask

Email is the oldest "message to a human". Two protocols: SMTP sends (delivers) mail from your app to a mail server; IMAP (and POP3) lets a mail program read it. Mixing these up is a classic exam trick: "SMTP retrieves email" is false.

MailHog for local testing

Never test with real addresses. MailHog is a fake mail server: it listens for SMTP on port 1025 and shows every message in a web page on port 8025 (http://localhost:8025). Download its single executable, run it, and point Flask-Mail at it. Nothing is delivered to the outside world.

Flask-Mail

pip install flask-mail

from flask import Flask
from flask_mail import Mail, Message

app = Flask(__name__)
app.config.update(
    MAIL_SERVER="localhost",
    MAIL_PORT=1025,            # MailHog's SMTP port
    MAIL_USE_TLS=False,
    MAIL_USE_SSL=False,
    MAIL_DEFAULT_SENDER="noreply@quizmaster.local",
)
mail = Mail(app)

def send_reminder(to, name):
    msg = Message(
        subject="Your daily reminder",
        recipients=[to],
        body=f"Hi {name}, you have not visited today.",              # plain-text part
        html=f"<h2>Hi {name}</h2><p>You have <b>not</b> visited today.</p>",  # HTML part
    )
    mail.send(msg)

with app.app_context():
    send_reminder("student@example.com", "Asha")

With MailHog running, the message appears in its inbox page. body is the plain text version, html the rich one; mail programs show the HTML if they can. Flask-Mail needs an application context (inside a view, or with app.app_context() in a script or Celery task). For real Gmail you would use smtp.gmail.com, port 587, MAIL_USE_TLS=True, and an app password kept in an environment variable (never in code). The standard library smtplib does the same job with more lines.

Rendering the HTML body from a Jinja template keeps the code clean:

from flask import render_template
html = render_template("email/reminder.html", name="Asha", pending=3)

Daily reminders and monthly reports with Celery beat

Sending 500 mails inside a request would freeze the page, so email goes in a Celery task, and Celery beat (the scheduler) starts it at fixed times.

from celery import Celery
from celery.schedules import crontab

celery = Celery("app", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")

@celery.task
def daily_reminder():
    # in a real app: query users who have not visited, then mail each
    return "reminders sent"

@celery.task
def monthly_report():
    return "reports sent"

celery.conf.beat_schedule = {
    "daily-6pm": {"task": "app.daily_reminder", "schedule": crontab(hour=18, minute=0)},
    "monthly-1st": {"task": "app.monthly_report",
                    "schedule": crontab(day_of_month=1, hour=8, minute=0)},
}
celery.conf.timezone = "Asia/Kolkata"

Run three processes: Redis, a worker (celery -A app.celery worker -l info; on Windows add -P solo) and beat (celery -A app.celery beat -l info). The task name defaults to module.function, here app.daily_reminder because the file is app.py. A crontab(hour=18, minute=0) means "every day at 18:00".

PDF reports and CSV exports

A monthly report is usually HTML turned into a PDF. The pipeline: data, then Jinja2 template to HTML, then a library renders the PDF, then the PDF is attached to the email.

from flask import render_template
from weasyprint import HTML        # pip install weasyprint (pdfkit needs the wkhtmltopdf program instead)

def make_pdf(user):
    html = render_template("report.html", user=user, stats=compute_stats(user))
    HTML(string=html).write_pdf(f"reports/{user.id}.pdf")

Attach it with msg.attach("report.pdf", "application/pdf", pdf_bytes).

A CSV export needs no extra library:

import csv, io
from flask import Response

@app.get("/export.csv")
def export_csv():
    out = io.StringIO()
    w = csv.writer(out)
    w.writerow(["id", "title", "score"])
    w.writerow([1, "Quiz 1", 8])
    return Response(out.getvalue(), mimetype="text/csv",
                    headers={"Content-Disposition": "attachment; filename=export.csv"})

Content-Disposition: attachment makes the browser download the file. For big exports, run it as a Celery task (it can take 20 to 40 seconds), then tell the page when it is done. The page should poll the task state every few seconds, or listen via SSE; guessing "wait 41 seconds with setTimeout" is wrong because the time varies.

Idea 5: Security and design of messaging endpoints

Webhook secrets and SMTP passwords must never reach the browser. This week's exam also showed a login example mixing three mistakes: building SQL by string formatting (SQL injection), storing and comparing plain-text passwords, and returning no token. The right answer there was "all of these". Use parameterised queries, store password hashes, and return a token (see the earlier token weeks).

Exam radar

Quick quiz

Viva: say it out loud

Recap

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 51 real questions on this week   All End Term papers