Posted inAI

Automating Technical Documentation with LLMs and GitHub Actions

AI tutorial - IT technology blog
AI tutorial - IT technology blog

The 2 AM Documentation Crisis

At 2 AM on a Tuesday, the production hotfix finally went live. We had just refactored a critical authentication module, changing three major API endpoints to resolve a race condition. The code worked perfectly. However, by 9 AM, I had 14 unread Slack messages from the frontend team. The complaints were consistent: ‘The API is returning 404s,’ ‘The README still shows the old auth header,’ and ‘Where is the changelog for this release?’

I realized then that while our CI/CD pipeline for code was state-of-the-art, our documentation was essentially manual. We relied on developer memory to update Markdown files. During high-pressure deployments, documentation is always the first casualty. This ‘documentation rot’ does more than just annoy colleagues; it creates integration bugs and slows down the entire engineering organization.

Root Cause: Why Your Docs Are Always Outdated

After that incident, I analyzed why our process failed. The issues weren’t about laziness, but rather the friction inherent in traditional workflows:

  • Cognitive Load: After finishing a complex feature, developers struggle to switch from ‘logic mode’ to ‘technical writing mode.’
  • Manual Synchronization: Changing a function signature in auth.py requires a manual search-and-replace across README.md and API_DOCS.md.
  • Lack of Validation: Pre-commit hooks can check for syntax errors, but they can’t verify if an English paragraph accurately reflects a logic change in Go or Python.

Tools like JSDoc or Doxygen offer some relief, but they often produce dry, robotic output. They miss the ‘why’ behind a change. Furthermore, they require a level of commenting discipline that most fast-moving teams simply can’t maintain during a sprint.

Comparing Approaches: Manual vs. Traditional vs. LLM

Before building a custom solution, I evaluated the three primary ways to handle technical writing:

  1. Manual Updates: These are accurate if done correctly, but they offer zero reliability and high friction.
  2. Static Generators (Swagger/Doxygen): These tools are reliable for structure. Unfortunately, they struggle with high-level explanations and require developers to write documentation inside the code anyway.
  3. LLM-Powered Automation: This uses Large Language Models to analyze code diffs and generate human-readable documentation. It captures the intent of the change, not just the syntax.

The choice was clear. By integrating an LLM into our GitHub workflow, we could automate the tedious parts of documentation while maintaining high-quality, readable content.

The Solution: A GitHub Actions + LLM Pipeline

The most effective strategy is to trigger a documentation sync every time a Pull Request merges into the main branch. I have deployed this in production environments, and the results are consistently stable. The pipeline follows a straightforward logic: detect changes, extract context, prompt the LLM, update files, and commit the changes.

Step 1: Setting up the GitHub Action

We need a workflow that triggers on pushes to the main branch. This workflow requires repository access and an API key for your chosen LLM (OpenAI, Claude, or Gemini).

name: Auto-Doc Generator

on:
  push:
    branches:
      - main

jobs:
  update-docs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: |
          pip install openai gitpython

      - name: Run Doc Generator Script
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python scripts/generate_docs.py

      - name: Commit and Push Changes
        run: |
          git config --global user.name 'github-actions[bot]'
          git config --global user.email 'github-actions[bot]@users.noreply.github.com'
          git add README.md API_DOCS.md CHANGELOG.md
          git commit -m "docs: automated update via LLM pipeline [skip ci]" || echo "No changes to commit"
          git push

Step 2: The Core Logic Script

This Python script acts as the engine. It identifies changes in the last commit and instructs the LLM to update the relevant files. Using GitPython, we can isolate the exact diff between the current commit and its predecessor.

import os
from openai import OpenAI
from git import Repo

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def get_code_diff():
    repo = Repo(".")
    diff = repo.git.diff('HEAD~1', 'HEAD')
    return diff

def update_file(filename, prompt_context):
    with open(filename, "r") as f:
        current_content = f.read()

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": f"You are a technical writer. Update the {filename} based on the code changes. Keep the existing tone and formatting."},
            {"role": "user", "content": f"Current {filename}:\n{current_content}\n\nCode Changes:\n{prompt_context}"}
        ]
    )
    return response.choices[0].message.content

def main():
    diff = get_code_diff()
    if not diff:
        return

    # Update README and Changelog
    for doc_file in ["README.md", "CHANGELOG.md"]:
        updated_content = update_file(doc_file, diff)
        with open(doc_file, "w") as f:
            f.write(updated_content)

if __name__ == "__main__":
    main()

Step 3: Engineering Reliable Prompts

The quality of your documentation depends entirely on your prompt. If you simply ask the LLM to ‘update the docs,’ it might hallucinate or delete critical sections. Specificity is vital. For CHANGELOG.md, I use a prompt that enforces the ‘Keep a Changelog’ standard (Added, Changed, Deprecated, Removed, Fixed).

Providing the LLM with the existing file content is a non-negotiable step. This context helps the model maintain consistent tone and formatting. Without it, your documentation will eventually look like a disjointed patchwork of different writing styles.

Handling Edge Cases and Reliability

Initial implementations often hit two hurdles. First, massive diffs can exceed the 128k token limit of models like GPT-4o. To solve this, I modified the script to filter out irrelevant files, such as .css, .svg, or lockfiles, before sending the data to the API.

Second, you must avoid the ‘Infinite Loop.’ Since the GitHub Action commits changes back to the repository, it could theoretically trigger itself indefinitely. Adding [skip ci] to the commit message is the industry-standard way to prevent this recursion.

The Result: Documentation That Evolves with Code

Since moving to this automated pipeline, those 9 AM Slack panics have vanished. When a developer merges a PR, the documentation updates within roughly 90 seconds. The accuracy is surprisingly high. This is because the LLM analyzes the actual implementation rather than relying on what the developer intended to write.

Documentation is no longer a chore we dread at the end of a sprint. It has become a natural byproduct of our development process. If you are tired of documentation drift, offloading the heavy lifting to an LLM is a practical, high-impact solution for any engineering team.

Share:
Tags: