Django site

Django site#

The publishing layer for the corpus the News pipeline pipeline produces. A Django project run to search, visualize, alert on, and share open-source reporting across their team.

Django earns its place here because the heavy lifting (ORM, admin, auth, sessions, CSRF, middleware, migrations) is already built; write models, views, and templates against those rails rather than re-inventing them.

Architecture#

Django follows the MVT pattern (Model, View, Template), the Django spin on MVC. A request flows top-to-bottom through a middleware stack, gets matched to a URL route, the view queries models, renders a template, the response unwinds back up the middleware stack.

        flowchart LR
    A[Client] -->|HTTP| B[WSGI/ASGI server]
    B --> C[Middleware stack]
    C --> D[URL router]
    D --> E[View]
    E -->|ORM| F[(Database)]
    E --> G[Template]
    G -->|HTML/JSON| C
    C -->|Response| A

Layer

Responsibility

WSGI / ASGI server

gunicorn (sync) or uvicorn (async). Production reverse-proxied behind nginx or caddy.

Middleware stack

Auth, sessions, CSRF, security headers, gzip, logging. Order matters; outermost wraps everything.

URL router

Maps URL pattern to view callable.

View

Function or class. Reads request, queries models, returns response (HTML, JSON, redirect, file).

Template

Django template language. Server-rendered HTML with inheritance and includes.

Model

ORM declaration. Defines schema, indexes, constraints, querysets.

Migrations

Generated from model changes (makemigrations); applied in order (migrate).

Project layout#

The standard Django layout, one project (the site) with multiple apps (each a self-contained domain).

site/
├── manage.py
├── pyproject.toml
├── site/                    # the Django *project* (settings)
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   ├── wsgi.py
│   └── asgi.py
├── corpus/                  # an app
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── models.py
│   ├── views.py
│   ├── urls.py
│   ├── forms.py
│   ├── templates/corpus/
│   │   ├── base.html
│   │   ├── search.html
│   │   └── article_detail.html
│   └── migrations/
├── alerts/                  # another app
│   ├── models.py
│   ├── tasks.py             # celery tasks (if used)
│   └── views.py
└── static/

Models#

The data the corpus app stores. Indexes named explicitly because operator queries filter on publish_date and sentiment.

# corpus/models.py
from django.db import models
from django.contrib.auth import get_user_model

class Article(models.Model):
    url           = models.URLField(max_length=2048, unique=True)
    title         = models.CharField(max_length=512)
    text          = models.TextField()
    fetched_at    = models.DateTimeField()
    publish_date  = models.DateTimeField(null=True, blank=True)
    sentiment     = models.FloatField()
    organizations = models.JSONField(default=list)
    persons       = models.JSONField(default=list)

    class Meta:
        indexes = [
            models.Index(fields=["publish_date"]),
            models.Index(fields=["sentiment"]),
        ]

    def __str__(self):
        return self.title

class SavedQuery(models.Model):
    owner       = models.ForeignKey(get_user_model(),
                                    on_delete=models.CASCADE)
    name        = models.CharField(max_length=200)
    q           = models.CharField(max_length=512)
    entity      = models.CharField(max_length=200, blank=True)
    min_sent    = models.FloatField(null=True, blank=True)
    max_sent    = models.FloatField(null=True, blank=True)
    created_at  = models.DateTimeField(auto_now_add=True)

Views#

Function-based views for simple endpoints, class-based for listing / detail. The corpus app exposes search and detail.

# corpus/views.py
from django.shortcuts import render, get_object_or_404
from django.db.models import Q
from .models import Article

def search(request):
    q = request.GET.get("q", "")
    entity = request.GET.get("entity", "")
    results = Article.objects.all()
    if q:
        results = results.filter(
            Q(title__icontains=q) | Q(text__icontains=q)
        )
    if entity:
        results = results.filter(
            Q(organizations__icontains=entity) |
            Q(persons__icontains=entity)
        )
    return render(request, "corpus/search.html", {
        "q": q, "entity": entity,
        "results": results.order_by("-publish_date")[:50],
    })

def article_detail(request, pk):
    art = get_object_or_404(Article, pk=pk)
    return render(request, "corpus/article_detail.html",
                  {"article": art})

# corpus/urls.py
from django.urls import path
from . import views

urlpatterns = [
    path("", views.search, name="search"),
    path("article/<int:pk>/", views.article_detail, name="article"),
]

Templates#

Server-rendered HTML with inheritance.

{# corpus/templates/corpus/base.html #}
<!doctype html>
<html>
  <head><title>{% block title %}News{% endblock %}</title></head>
  <body>
    <header><a href="{% url 'search' %}">Search</a></header>
    {% block content %}{% endblock %}
  </body>
</html>

Admin#

Django’s built-in admin gives the operator a CRUD UI for free. Register the model, add list-display and filter hints.

# corpus/admin.py
from django.contrib import admin
from .models import Article, SavedQuery

@admin.register(Article)
class ArticleAdmin(admin.ModelAdmin):
    list_display  = ("title", "publish_date", "sentiment")
    list_filter   = ("publish_date",)
    search_fields = ("title", "text")
    ordering      = ("-publish_date",)

admin.site.register(SavedQuery)

Auth#

Django ships sessions, password hashing (PBKDF2 or Argon2 if configured), permissions, groups, and a @login_required decorator. Add django-allauth for OAuth and SSO when the team authenticates against an IdP.

from django.contrib.auth.decorators import login_required

@login_required
def search(request):
    ...

Alerts#

Per-user saved queries that run on a schedule and email or webhook the user when new articles match.

        sequenceDiagram
    participant Cron as Celery beat
    participant Worker as Celery worker
    participant DB as Database
    participant SMTP as SMTP / webhook

    Cron->>Worker: tick (every 5m)
    Worker->>DB: load SavedQuery rows
    loop for each SavedQuery
        Worker->>DB: query Article matching filters
        DB-->>Worker: new matches since last run
        Worker->>SMTP: send email / POST webhook
    end

The worker process runs Celery against Redis (broker) or PostgreSQL (with django-celery-beat). See Frameworks for the Celery side.

Common Tasks#

Bootstrap a Django project.

$ uv add django
$ uv run django-admin startproject site .
$ uv run python manage.py startapp corpus
$ uv run python manage.py migrate

Make and apply migrations after a model change.

$ uv run python manage.py makemigrations
$ uv run python manage.py migrate

Create a superuser for the admin.

$ uv run python manage.py createsuperuser

Run the dev server.

$ uv run python manage.py runserver 0.0.0.0:8000

Run a one-off shell against the project (ORM available).

$ uv run python manage.py shell
>>> from corpus.models import Article
>>> Article.objects.filter(sentiment__lt=-0.5).count()

Load the tabulated corpus into the database.

$ uv run python manage.py load_corpus ../news/data/tabulated/corpus.parquet

Run tests.

$ uv run python manage.py test

References#

Contents