Building My Second Brain with AI

2026-01-25

A comprehensive guide to understanding how this personal knowledge management system works, why it works this way, and what we learned building it.

Contents

1. What Is This Thing, Anyway?
2. How Big Is This Thing?
3. The Technology Stack
4. The AI Classification System
5. Slack Integration
6. The People System
7. Lessons Learned
8. Architecture Decisions
9. Closing Thoughts
10. Inspiration

What Is This Thing, Anyway?

Imagine having a brilliant assistant who: - Catches every thought you throw at them via Slack, email, or a web app - Instantly knows whether that thought is a task, a note, a meeting summary, or a personal reflection - Files it in exactly the right place without you having to think about it - Remembers everything and can surface it when you need it

That’s what we built. It’s a personal knowledge management system designed around one core principle: capture should be frictionless, organization should be automatic.

The system has four main jobs:

1. Capture - Frictionless input from anywhere: iPhone action button, keyboard shortcut on the laptop, Slack, web, email, meeting recordings
2. Classify - AI uses expansive context to figure out what it is and where it belongs
3. Surface - Show the right information at the right time
4. Build Trust - Comprehensive logs and self-corrections so you can actually trust the system

But as delicious as frictionless capture is, the resulting mountain of items is still there. That’s where the human work begins: morning routines to limit what a successful day looks like, weekly reviews to ask if you’re living your standards, and a manageable active project list—so you can say no to the seductive easy tasks and focus on the hard things

How Big Is This Thing?

One-liner: A ~35K LOC personal productivity system with a Python/Flask backend, 73 API endpoints, 12 database tables, AI-powered classification via Claude, and integrations with Slack, Google Calendar, Fathom, and Readwise. Roughly the complexity of a seed-stage B2B SaaS product, built for a single power user.

Code Metrics

Database

API Surface

External Integrations

Operations

The Big Picture: How Everything Connects

                    ┌─────────────────────────────────────┐
                    │         YOU (The User)              │
                    └───────────┬─────────────────────────┘
                                │
        ┌───────────────────────┼───────────────────────────┐
        │                       │                           │
        ▼                       ▼                           ▼
┌───────────────┐      ┌───────────────┐      ┌─────────────────┐
│    Slack      │      │   Web App     │      │ Fathom Meetings │
│ #kuty-capture │      │ localhost:5001│      │   (Webhooks)    │
└───────┬───────┘      └───────┬───────┘      └────────┬────────┘
        │                      │                       │
        └──────────────────────┼───────────────────────┘
                               │
                               ▼
                    ┌─────────────────────────────────────┐
                    │      AI Classification (Claude)     │
                    │   "Is this a task? A note? About    │
                    │    who/what project?"               │
                    └───────────────┬─────────────────────┘
                                    │
                                    ▼
                    ┌─────────────────────────────────────┐
                    │        Supabase (PostgreSQL)        │
                    │   THE SINGLE SOURCE OF TRUTH        │
                    │                                     │
                    │   tasks │ projects │ notes          │
                    │   people │ reflections │ media      │
                    │   transcripts │ change_log          │
                    └───────────────┬─────────────────────┘
                                    │
                    ┌───────────────┼───────────────────────┐
                    │               │                       │
                    ▼               ▼                       ▼
           ┌────────────┐  ┌────────────────┐    ┌──────────────┐
           │  Web UI    │  │   Tasks.md     │    │   Slack      │
           │  (Flask)   │  │  (Generated)   │    │  Digests     │
           └────────────┘  └────────────────┘    └──────────────┘

The key insight: The database is the brain, everything else is just a window into it.

The Technology Stack: Why We Chose What We Chose

The Backend: Python + Flask + Supabase

Why Python? It’s the language that gets out of your way. When you’re building a personal productivity system, you want to move fast and iterate. Python lets you write db.get_open_tasks() and know exactly what it does.

Why Flask? It’s the un-framework framework. Django would have forced us into its way of thinking about things. Flask says “here’s a way to handle HTTP requests, figure out the rest yourself.” For a personal tool that’s constantly evolving, that flexibility is gold.

Why Supabase? Three reasons: 1. PostgreSQL under the hood - A real database, not a toy 2. Hosted - No DevOps headaches 3. Instant API - The Python client lets us write db.client.table('tasks').select('*').eq('status', 'open') and it just works

Analogy: Supabase is like having a database administrator on call 24/7 who never complains about your schema changes.

The Frontend: Jinja2 + HTMX + Alpine.js + Tailwind CSS

This combination might look unusual, but it’s the sweet spot for productivity apps.

Jinja2 templates - Server-side rendering means the page loads with data already there. No “loading…” spinners, no client-side state to manage.

HTMX - This is the secret weapon. Instead of building a full React app with state management, API calls, and loading states, we write:

<button hx-post="/api/task/123/toggle"
        hx-target="closest .task-row"
        hx-swap="outerHTML">
    ✓ Complete
</button>

Click the button, the server handles the logic, returns the new HTML, and HTMX swaps it in. No JavaScript state to manage. No Redux. No “why is my component re-rendering?”

Analogy: HTMX is like the Slack message protocol but for your HTML. Instead of fetching data and transforming it into UI, you just ask the server for the UI directly.

Alpine.js - For the interactive bits that need to happen purely on the client (expanding a dropdown, showing/hiding a form), Alpine.js gives you just enough JavaScript without the React tax.

Tailwind CSS - No more naming CSS classes. text-sm text-gray-600 hover:text-blue-600 is self-documenting.

The Database Schema: A Map of Your Mind

The database has evolved to model how knowledge actually works:

Core Entities

The Magic Table: change_log

Every single change to the database gets logged:

INSERT INTO change_log (entity_type, entity_id, action, field_name, old_value, new_value, source)
VALUES ('task', 123, 'update', 'status', 'open', 'complete', 'web');

This seems like overkill until you want to: - Show “what did I accomplish today?” - Track how long tasks took from creation to completion - Debug “wait, who changed this?” - Build the Future Log activity metrics

Lesson: If you’re building a productivity system, log everything. Storage is cheap, context is priceless.

The AI Classification System: Teaching Claude to Think Like You

Here’s where it gets interesting. When you type “call Jesus about the video series” into Slack, how does the system know: 1. This is a task (not a note, not a reflection) 2. It probably belongs to the AB-video-series project 3. It involves Jesus (a person in your database)

The Classification Prompt

The classifier sends your input to Claude with a carefully crafted prompt that includes: - The raw text you typed - All your active projects (names, areas, outcomes) - All the people in your database (names, tags, context)

This is called dynamic context injection. Instead of training a custom model, we give Claude just-in-time context about YOUR world.

CURRENT CONTEXT:

Active Projects:
- AB-video-series (AB): Launch new video content platform
- LU-client-onboarding (LU): Streamline client setup process
...

Known People:
- Jesus [AB]: VP of Product at AlephBeta
- Tikva [AB, LU]: CFO at both organizations
...

Use the above context to match projects and people accurately.

Confidence Scoring: Knowing When You Don’t Know

Every classification comes with two confidence scores:

When either score drops below 0.6, the item goes into a review queue instead of being auto-filed. You’ll see it later and can fix the classification.

Analogy: It’s like having an assistant who says “I filed this as a task for the video project, but I’m only 40% sure - want to double-check?”

Manual Override Markers

Sometimes you just know: - *reflection - This is definitely a reflection - -call mom - This is definitely a task - @AB-video-series - This definitely goes to this project

Manual markers skip AI classification entirely. The system respects explicit instructions.

The Web UI: What Makes It Feel Good

The Philosophy: Server-Side First

Every page loads with data already rendered. No spinners, no “fetching…”, no client-side hydration. The HTML that comes over the wire is the final UI.

When you toggle a task checkbox: 1. HTMX sends a POST to /api/task/123/toggle 2. The server updates the database 3. The server re-renders just the task row HTML 4. HTMX swaps out the old row for the new one 5. Total round-trip: ~50-100ms

The Pattern: Mutation → Render → Swap

Every action follows this pattern:

@app.route('/api/task/<int:task_id>/toggle', methods=['POST'])
def toggle_task(task_id):
    # 1. Mutate
    db.toggle_task(task_id)

    # 2. Re-fetch
    task = db.get_task(task_id)

    # 3. Render
    response = make_response(render_template('components/task_row.html', task=task))

    # 4. Trigger any cascading refreshes
    response.headers['HX-Trigger'] = 'refreshTasks'
    return response

Component Architecture

The UI is built from reusable Jinja2 components:

web/templates/components/
├── task_row.html        # A single task with checkbox, edit, move, delete
├── note_item.html       # A note with content, attachments, edit mode
├── project_selector.html # Filterable dropdown with inline creation
├── working_note_editor.html  # Quill rich text editor
└── ...

Each component is self-contained and can be swapped via HTMX without touching the rest of the page.

Slack Integration: The Capture Layer

Why Slack?

Slack is already open. Slack is where you’re already thinking. Friction is the enemy of capture.

The #kuty-capture channel becomes a trusted inbox. Type anything, hit enter, done. The system figures out the rest.

Socket Mode: No Servers Needed

Traditional Slack apps need a public URL for webhooks. Socket Mode uses WebSockets to connect to Slack directly. The script runs on your laptop and maintains a persistent connection.

# Instead of: POST from Slack to https://your-server.com/webhook
# We get: WebSocket connection from laptop to Slack

This means: - No public URLs to maintain - No ngrok or Cloudflare tunnels for development - Works from anywhere, even behind firewalls

File Handling: The Tricky Part

When you share an image in Slack, the process is: 1. Slack hosts the file at a private URL 2. We download it using the Slack API 3. We upload it to Supabase Storage (our own private bucket) 4. We generate a signed URL (valid for 1 year) 5. The markdown gets ![image.jpg](signed-url)

Why not just use Slack’s URL? Because Slack URLs expire, require authentication, and break when you leave the workspace. Our URLs are ours forever.

The Fix Command: Correcting Mistakes

When AI gets it wrong (and it will), you reply in the thread:

fix: task @AB-video-series

The system: 1. Removes the original entry 2. Re-classifies with your explicit instructions 3. Files it correctly 4. Updates the activity log

The People System: Your Relationship Memory

The Problem It Solves

You meet someone at a conference. They mentioned their daughter just started college. Three months later, you email them and… you’ve completely forgotten the context. Who were they again?

The People system is a lightweight CRM that captures relationship context, not just contact info.

What Gets Stored

AI-Powered Classification

When a new contact comes in via email capture, the classifier doesn’t just extract contact info - it figures out which area of your life this person belongs to:

# The classifier looks at:
# - Email domain (@alephbeta.org → AB, @lumenalta.com → LU)
# - Email subject/body keywords
# - Explicit mentions of projects or areas

result = classify_person_area(
    name="John Smith",
    email="john@acme-capital.com",
    email_subject="Re: Series A Discussion",
    email_body="Great meeting you at the conference..."
)
# → area_code: "FIN", confidence: 0.85, context: "Potential investor from Acme Capital"

When confidence is low (< 0.6), a yellow ? badge appears in the UI - a gentle nudge to review and correct.

The “Needs Phone” Flag

Some contacts have email but not phone numbers. The system tracks this with a needs_phone_followup flag, and you can filter to see everyone who needs phone capture.

Why this matters: Phone numbers are gold for relationship-building. Email is easy to ignore; a call isn’t.

Readwise Integration: Morning Inspiration

The Idea

You’ve highlighted hundreds of passages in articles over the years. Most sit forgotten in Readwise. What if one random highlight showed up on your home page every morning?

How It Works

def get_random_readwise_highlight():
    """Fetch a random highlight from Readwise articles."""
    response = requests.get(
        'https://readwise.io/api/v2/highlights/',
        headers={'Authorization': f'Token {readwise_token}'},
        params={'category': 'articles', 'page_size': 100}
    )
    highlights = response.json().get('results', [])
    return random.choice(highlights)

The home page shows: - The highlight text - The article title and author - A link to the source

Analogy: It’s like having a wise friend who reminds you of something you once thought was important enough to highlight.

Design Decision: Fail Silently

If Readwise is down or the token is missing, the highlight just doesn’t appear. No error messages, no broken UI. The highlight is a “nice to have,” not critical functionality.

except Exception:
    pass  # Fail silently - highlight is optional

Reflection Prompts: Structured Introspection

The Problem

You open the reflection form and… blank. What should you write about? The empty text box is intimidating.

The Solution: Time-Based Prompts

The system shows a different reflection prompt depending on the time of day:

Morning Prompts (before noon): - “What has to happen today for it to be a good day?” - “If we meet a year from now, what will I want to be on track to celebrate?” - “What is okay to say no to today?” - “What am I doing when I feel most alive, and how can I do a little of that today?” - “What single sentence describes who I choose to be today?”

Evening Prompts (noon and after): - “What was my story from today—the moment that made today different from any other day?” - “Where did I take care of what matters well today, and where did I miss it?” - “What problems am I facing now that I would have once begged to have?” - “What did I learn or notice today that I don’t want to lose?” - “If I died tonight, what would I regret not doing more of—and what tiny move can I make toward that tomorrow?”

The prompt is randomly selected each page load. It appears in a purple banner above the reflection form.

Why this works: A good question is a thinking tool. Instead of “what should I reflect on?”, you’re responding to a specific, thought-provoking prompt.

The Future Log: Monthly Review & Project Health

The Concept

Borrowed from Bullet Journal methodology, the Future Log is where you step back from daily tasks and look at the bigger picture: - Which areas of your life got attention this month? - Which projects moved forward? - Which areas went dormant? - What decisions did you make?

The Layout

┌────────────────────────────┬────────────────────────────┐
│     Monthly Note           │     Activity Report        │
│                            │                            │
│  [Editable textarea]       │  ┌─ AB: AlephBeta ───────┐ │
│                            │  │ ✓ 12 tasks closed     │ │
│  Reflect on January...     │  │ 📝 5 notes written    │ │
│  What happened?            │  │                       │ │
│  What did you learn?       │  │ ▼ Role Standards      │ │
│                            │  │   "Focus on content   │ │
│                            │  │    quality over       │ │
│                            │  │    quantity..."       │ │
│                            │  │                       │ │
│                            │  │ Projects:             │ │
│                            │  │ • video-series [active]│ │
│                            │  │ • platform-v2  [soon] │ │
│                            │  └───────────────────────┘ │
│                            │                            │
│                            │  ┌─ LU: Lumenalta ──────┐ │
│                            │  │ [dormant] ⚠️          │ │
│                            │  └───────────────────────┘ │
└────────────────────────────┴────────────────────────────┘

Key Features

Activity Metrics from change_log: - Tasks closed (status changed to ‘complete’) - Notes written (new note entries) - Dormant detection (no activity in the period)

Role Standards: Each area has an editable “standards” field: “What does ‘good’ look like for your role in this area?”

This is where you capture things like: - “For AlephBeta: Ship one video per week, respond to community within 24 hours” - “For Family: Date night weekly, one-on-one with each kid monthly”

The standards are shown in the area card so you can compare your activity against your intentions.

Period Navigation: Toggle between Month / Quarter / Year views to see activity at different timescales.

Decisions Tab: When you complete a project, you write a closing note. These appear in the Decisions tab - a record of what you decided and why.

Why this matters: Most productivity systems are great at capturing what to do. They’re terrible at capturing why you did it. The Decisions tab is your institutional memory.

The Fathom Integration: Automatic Meeting Notes

The Architecture

Fathom finishes recording
        │
        ▼ HTTP POST
┌─────────────────────────┐
│  Supabase Edge Function │  (Deno/TypeScript)
│   fathom-webhook        │
└───────────┬─────────────┘
            │
    ┌───────┼───────┐
    ▼       ▼       ▼
┌──────┐ ┌──────┐ ┌──────────┐
│  DB  │ │Slack │ │Background│
│      │ │Notif │ │Enrichment│
└──────┘ └──────┘ └──────────┘

Why a Supabase Edge Function?

The webhook needs to be always available (not running on your laptop). Supabase Edge Functions give us: - Serverless deployment (no infrastructure) - Built-in Supabase database access - TypeScript for type safety - Free tier that covers our usage

The Enrichment Problem

Fathom only knows about attendees from its connected calendar. If you have multiple calendars (personal + work), Fathom might miss people.

Solution: enrich-transcripts.py runs every 5 minutes, queries Google Calendar for matching events, and updates the attendees list.

This is a recurring pattern: Accept imperfect data, enrich it asynchronously.

Lessons Learned: Bugs, Pitfalls, and Hard-Won Wisdom

Lesson 1: The N+1 Query Problem

The Bug:

# WRONG: This makes N database calls
for project in projects:
    task_count = db.client.table('tasks').select('id', count='exact').eq('project_id', project['id']).execute()
    project['open_tasks'] = task_count.count

With 20 projects, this makes 20 database round-trips. Over the network. Each one taking 50-100ms. Your page now takes 2 seconds to load.

The Fix:

# RIGHT: One database call, then map in Python
task_counts = db.client.table('tasks').select('project_id').eq('status', 'open').execute()
counts_by_project = {}
for task in task_counts.data:
    pid = task['project_id']
    counts_by_project[pid] = counts_by_project.get(pid, 0) + 1

for project in projects:
    project['open_tasks'] = counts_by_project.get(project['id'], 0)

The Rule: Fetch all related data in ONE query, transform in code.

Lesson 2: The |tojson Escaping Nightmare

The Bug:

<!-- WRONG: Double quotes break HTML -->
<div x-data="{ text: {{ value|tojson }} }">

If value = "Hello's world", this renders as:

<div x-data="{ text: "Hello\u0027s world" }">

HTML parsing breaks at the second quote.

The Fix:

<!-- CORRECT: Single quotes wrap the attribute -->
<div x-data='{ "text": {{ value|default("", true)|tojson }} }'>

The Rule: Always use single quotes for HTML attributes containing |tojson output.

Lesson 3: Hidden Required Fields Break Silently

The Bug:

<div x-show="creating">
    <input name="name" required>
</div>

When creating = false, the div is hidden but the input still exists. HTML5 validation sees a required field that’s empty and silently prevents form submission.

The Fix:

<div x-show="creating">
    <input x-model="name">
    <button :disabled="!name">Submit</button>
</div>

The Rule: Never use required on fields in collapsible sections. Use :disabled on buttons instead.

Lesson 4: The Flask Debug Reloader Memory Leak

The Bug: Flask’s debug reloader creates a new Python process when code changes. The Supabase client in the old process doesn’t properly close. The new process inherits a stale connection. Queries return empty results.

The Fix:

@app.before_request
def ensure_db_healthy():
    """Reset database client if returning empty data."""
    try:
        result = db.client.table('areas').select('code', count='exact').execute()
        if result.count == 0:
            db._client = None  # Reset lazy-loaded client
            # Retry
            result = db.client.table('areas').select('code', count='exact').execute()
    except Exception:
        db._client = None

The Rule: In development, expect hot-reloading to break database connections. Build auto-recovery.

Lesson 5: HTMX Events Don’t Bubble Like You Think

The Bug:

<!-- Component returns HX-Trigger: refreshTasks -->
<div hx-get="/api/tasks" hx-trigger="refreshTasks from:body">

You’d expect the refreshTasks event to trigger a refresh. It doesn’t. HX-Trigger fires on the element that made the request, not on body.

The Fix: For inline edits, use hx-swap="outerHTML" and return the updated component. For list refreshes, use htmx.trigger(document.body, 'refreshTasks') in JavaScript.

The Rule: HTMX is simple until it isn’t. Read the docs on event bubbling carefully.

Lesson 6: The Lazy-Loaded Client Pattern

The Pattern:

class JournalDatabase:
    def __init__(self):
        self._client: Optional[Client] = None

    @property
    def client(self) -> Client:
        """Create client on first access."""
        if self._client is None:
            self._client = create_client(self._url, self._key)
        return self._client

This delays connection until first use, allows recovery by setting _client = None, and keeps initialization fast.

The Rule: For external connections, lazy-load and plan for recovery.

Lesson 7: Background Threads for Slow Operations

The Bug: Toggling a task checkbox took 2 seconds because it regenerated the Tasks.md report synchronously.

The Fix:

@app.route('/api/task/<int:task_id>/toggle', methods=['POST'])
def toggle_task(task_id):
    db.toggle_task(task_id)
    task = db.get_task(task_id)

    # Regenerate report in background (don't block response)
    threading.Thread(target=regenerate_tasks_report, daemon=True).start()

    return render_template('components/task_row.html', task=task)

The Rule: Never make the user wait for operations they don’t care about.

Lesson 8: Soft Deletes Save Lives

The Pattern:

def delete_task(self, task_id: int):
    # WRONG: self.client.table('tasks').delete().eq('id', task_id).execute()

    # RIGHT:
    self.client.table('tasks').update({
        'deleted_at': datetime.now().isoformat()
    }).eq('id', task_id).execute()

Every query includes .is_('deleted_at', 'null'). Deleted items are invisible but recoverable.

The Rule: Never hard-delete in a productivity system. You will want that data back.

How Good Engineers Think About This Stuff

Start with the Data Model

Before writing any UI code, we defined what the data looks like. Projects have outcomes. Tasks have statuses. Changes get logged. The schema is the contract.

Make the Happy Path Obvious

When you type in Slack, the most common outcome is: AI classifies correctly, task appears in the right project. That path should be automatic and instant. The edge cases (fix commands, review queues) exist but don’t clutter the main flow.

Audit Everything

The change_log table seemed like overkill at first. Now it powers: - Activity feeds - Future Log metrics - Debugging - Accountability

You can’t add logging retroactively. Build it in from day one.

Fail Gracefully

When the AI classifier can’t make a confident decision, it doesn’t guess. It queues the item for review and tells you why. Better to surface uncertainty than to file things wrong.

Prefer Explicit Over Clever

The manual markers (*, -, @PROJECT) let you override AI. The fix command lets you correct mistakes. The system doesn’t pretend to be smarter than you.

The Architecture Decisions We’re Proudest Of

1. Database as Source of Truth

Early versions synced between markdown files and database. This was a nightmare: - Which version is authoritative? - What happens if they conflict? - How do you handle partial syncs?

Now: Database is truth. Markdown files are generated outputs (read-only). No conflicts possible.

2. HTMX Instead of React

We could have built a slick SPA with React, Redux, and 10,000 lines of TypeScript. Instead: - Templates render on the server - HTMX handles interactions - Alpine.js adds client-side sugar

The result: Faster development, simpler debugging, no build step, and a UI that feels just as snappy.

3. AI Classification with Confidence Scores

Binary classification (“this is a task”) would be wrong too often. Confidence scores let us: - Auto-file high-confidence items (speed) - Queue low-confidence items (accuracy) - Track classifier performance over time

4. Background Enrichment

Fathom gives us incomplete attendee lists. Google Calendar has the full list. Instead of blocking on this: - Accept Fathom’s data immediately - Run enrichment in the background - Update when better data is available

This pattern works for any “eventually consistent” scenario.

What’s Next?

The system is functional but not complete. Ideas for the future:

1. Email capture - Parse inbox for tasks and contacts (partially built)
2. Calendar integration - Show upcoming meetings alongside tasks
3. Mobile app - The web UI works on mobile, but a native app would be slicker
4. Better search - Full-text search across all entities
5. AI summarization - “What did I accomplish this week?”

Closing Thoughts

Building this system taught us that personal productivity tools are deeply personal. Off-the-shelf solutions force you into their model. Custom tools let you build the model around how YOU think.

The technology choices matter less than you’d think. What matters: - Fast capture (no friction) - Smart classification (less manual filing) - Reliable storage (nothing gets lost) - Flexible views (see what you need when you need it)

Everything else is implementation details.

I’d Love to Hear From You

If you’re building something similar—a personal knowledge system, an AI-powered productivity tool, or just experimenting with how to capture and organize your thoughts—I want to hear from you.

Not because I think my approach is right for everyone, but because the more we share what we’re learning, the faster we all get there.

Say hello.

Inspiration

This project was inspired by a YouTube video on building a second brain with AI. The core insight: capture should be frictionless, organization should be automatic.