Lessons Learned - ClaudeOps Development

This document captures critical lessons learned during the development and deployment of ClaudeOps. These hard-won insights will save you hours of debugging.

---

Critical: Shell Management

NEVER Delete Your Current Working Directory

The Problem:

If you cd into a directory and then delete it, the shell's working directory becomes invalid and ALL subsequent commands will fail.

Example of what NOT to do:

cd /tmp/test-directory
rm -rf /tmp/test-directory  # ❌ BREAKS THE SHELL
# Everything after this fails with "getcwd: cannot access parent directories"

What Happens:

• You get: pwd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
• Every subsequent command returns "Error"
• The shell is in an invalid state
• In automated environments (cron, CI/CD), this causes complete failure

Correct Approaches:

# Option 1: Use absolute paths (BEST)
rm -rf /tmp/test-directory  # ✅ SAFE - no cd needed

# Option 2: Change directory first, then delete
cd /home/claude && rm -rf /tmp/test-directory  # ✅ SAFE

# Option 3: Use subshell
(cd /tmp && rm -rf test-directory)  # ✅ SAFE - doesn't affect parent shell

# Option 4: Change to parent before deleting
cd /tmp
rm -rf test-directory  # ✅ SAFE - we're in parent, not child

Why This Matters:

In automated/cron environments, a broken shell state prevents all subsequent operations from working, making the entire automation fail silently.

Recovery:

If this happens in an interactive session, you can't recover the current shell. You must:

1. Exit the current shell
2. Start a new shell session
3. In automated systems, delegate to a subagent or new process

Encountered: 2025-10-02 during deployment testing

---

Cron Job Configuration

User Field Required in /etc/cron.d/

The Problem:

Cron jobs in /etc/cron.d/ require a user field that system crontabs don't need.

Wrong (system crontab format):

# /etc/cron.d/claudeops
0 */2 * * * /opt/claudeops/bin/claudeops-cron.sh  # ❌ FAILS - missing user field

Correct (/etc/cron.d/ format):

# /etc/cron.d/claudeops
0 */2 * * * root /opt/claudeops/bin/claudeops-cron.sh  # ✅ CORRECT - user field present

The Difference:

Crontab Type	Location	User Field	Example
System crontab	/etc/crontab	Required	0 * * * * root /path/to/script
Drop-in crontab	/etc/cron.d/*	Required	0 * * * * root /path/to/script
User crontab	crontab -e	Not allowed	0 * * * * /path/to/script

Symptoms of Missing User Field:

• Cron job doesn't run
• No error messages in logs
• grep CRON /var/log/syslog shows nothing
• Silent failure

Debugging:

# Check cron syntax
sudo crontab -T /etc/cron.d/claudeops

# Test cron expression
sudo run-parts --test /etc/cron.d/

# Monitor cron execution
sudo tail -f /var/log/syslog | grep CRON

# Verify cron service
sudo systemctl status cron

Best Practice:

Always specify the user when creating files in /etc/cron.d/:

# Good: Explicit user
0 */2 * * * root /opt/claudeops/bin/claudeops-cron.sh

# Also valid: Run as specific user
0 */2 * * * claudeops /opt/claudeops/bin/claudeops-cron.sh

Encountered: 2025-10-02 during cron setup

---

Git Configuration

Branch Naming: master vs main

The Problem:

Git defaults changed from master to main for new repositories, but many existing repos still use master.

Symptoms:

git push origin main
# error: src refspec main does not match any

Root Cause:

Your local branch is named main, but the remote expects master (or vice versa).

Solutions:

# Solution 1: Check what branch you're on
git branch
# * main

# Solution 2: Check remote branches
git branch -r
# origin/master

# Solution 3: Push to correct remote branch
git push origin main:master  # Push local 'main' to remote 'master'

# Solution 4: Rename local branch to match remote
git branch -m main master
git push origin master

# Solution 5: Set upstream properly
git push -u origin main  # Creates 'main' on remote if it doesn't exist

Best Practice:

Always verify branch names before pushing:

# Check current branch
git rev-parse --abbrev-ref HEAD

# Check remote branches
git ls-remote --heads origin

# Set default branch for new repos
git config --global init.defaultBranch main

Encountered: 2025-10-02 during GitHub integration

---

PATH Issues in Automated Environments

sudo and cron Don't Inherit User PATH

The Problem:

Commands that work in your shell fail in cron or when run via sudo because they can't find executables.

Example:

# Works in your shell
claudeops check

# Fails in cron
# /bin/sh: 1: claudeops: not found

Root Cause:

Different execution contexts have different PATH values:

Context	Typical PATH
User shell	/usr/local/bin:/usr/bin:/bin:/opt/claudeops/bin
Root shell	/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Cron	/usr/bin:/bin (minimal!)
Sudo	Resets to secure_path

Solutions:

# Solution 1: Use absolute paths (BEST for cron)
/opt/claudeops/bin/claudeops check

# Solution 2: Set PATH in cron file
PATH=/usr/local/bin:/usr/bin:/bin:/opt/claudeops/bin
0 */2 * * * root /opt/claudeops/bin/claudeops-cron.sh

# Solution 3: Set PATH in script
#!/bin/bash
export PATH="/opt/claudeops/bin:$PATH"
claudeops check

# Solution 4: Create symlink in standard location
sudo ln -s /opt/claudeops/bin/claudeops /usr/local/bin/claudeops

Debugging:

# Check PATH in cron
* * * * * root echo $PATH >> /tmp/cron-path.log

# Check what 'which' sees
which claudeops

# Verify executable permissions
ls -l /opt/claudeops/bin/claudeops

# Test as cron would execute
sudo -u root env -i /bin/sh -c 'claudeops check'

Best Practice for Cron:

Always use absolute paths in cron jobs:

# Good
0 */2 * * * root /opt/claudeops/bin/claudeops-cron.sh

# Bad (works locally, fails in cron)
0 */2 * * * root claudeops-cron.sh

Encountered: 2025-10-02 during cron debugging

---

Git Ownership and Safe Directories

fatal: detected dubious ownership in repository

The Problem:

Git refuses to operate on repositories not owned by the current user (security feature).

Symptoms:

sudo git status
# fatal: detected dubious ownership in repository at '/opt/claudeops'

Root Cause:

Git 2.35.2+ added security checks to prevent repository ownership attacks. If you're running git as user A in a directory owned by user B, git refuses to operate.

Common Scenarios:

1. Using sudo git in a user-owned directory
2. Running git in /opt/ directories owned by root
3. NFS/shared mounts with different ownership
4. Docker containers with mounted volumes

Solutions:

# Solution 1: Add to safe.directory (per repository)
sudo git config --global --add safe.directory /opt/claudeops

# Solution 2: Add wildcard (multiple repos)
sudo git config --global --add safe.directory '*'  # ⚠️ Less secure

# Solution 3: Fix ownership
sudo chown -R $(whoami):$(whoami) /opt/claudeops

# Solution 4: Use sudo -E to preserve environment
sudo -E git status

# Solution 5: Run as correct user
sudo -u claudeops git status

Security Implications:

Adding safe.directory bypasses security checks. Only do this for repositories you control.

Best Practice:

Match git user to repository owner:

# Good: Owner runs git
chown -R claudeops:claudeops /opt/claudeops
sudo -u claudeops git -C /opt/claudeops status

# Acceptable: Explicitly trust
git config --global --add safe.directory /opt/claudeops

# Bad: Disable security globally
git config --global safe.directory '*'  # ⚠️ Dangerous

Debugging:

# Check repository ownership
ls -la /opt/claudeops/.git

# Check current user
whoami

# Check git config
git config --global --get-all safe.directory

# Check git version (security feature added in 2.35.2)
git --version

Encountered: 2025-10-02 during Git log automation

---

System Service Configuration

Systemd Service Dependencies

The Problem:

Services start in the wrong order, causing failures when dependencies aren't ready.

Example:

# Bad: No dependency specification
[Unit]
Description=My App

[Service]
ExecStart=/usr/bin/myapp

[Install]
WantedBy=multi-user.target

App starts before PostgreSQL is ready → connection errors.

Solution:

Properly specify dependencies:

# Good: Explicit dependencies
[Unit]
Description=My App
After=network.target postgresql.service
Requires=postgresql.service

[Service]
ExecStart=/usr/bin/myapp
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Key Directives:

Directive	Meaning
After=	Start after this service (doesn't require it)
Before=	Start before this service
Requires=	This service is required (fails if it fails)
Wants=	This service is desired (doesn't fail if it fails)
BindsTo=	Like Requires, but also stops if dependency stops

Best Practices:

# Database-dependent app
After=network.target postgresql.service
Requires=postgresql.service

# Web server with database
After=network.target postgresql.service
Wants=postgresql.service  # Prefer Wants for optional deps

# Boot recovery service
After=network-online.target
Wants=network-online.target

Encountered: 2025-10-02 during boot recovery setup

---

Documentation and Communication

Markdown Formatting in Logs

The Lesson:

Human-readable logs are worth the effort. Claude excels at generating structured markdown reports.

Good Log:

# Health Check - 2025-10-02 14:00

## Status: 🟢 GREEN

### Services
- ✅ PostgreSQL: Running, 15 connections
- ✅ Frontend: Responding in 120ms

### Actions Taken
None - all systems nominal

Bad Log:

2025-10-02 14:00:00 [INFO] Health check started
2025-10-02 14:00:01 [INFO] PostgreSQL: OK
2025-10-02 14:00:02 [INFO] Frontend: OK
2025-10-02 14:00:03 [INFO] Health check complete

Why Markdown:

• Easy to read in terminal
• Renders nicely on GitHub
• Claude generates it naturally
• Hierarchical structure
• Supports code blocks, tables, lists

Best Practices:

• Use emoji for status (🟢🟡🔴)
• Include timestamps
• Explain reasoning
• Provide context from previous runs
• Make actionable recommendations

Encountered: Throughout development

---

Environment Variables in Cron

Cron Doesn't Have Your Environment

The Problem:

Scripts that work manually fail in cron because environment variables are missing.

Symptoms:

# Works manually
./my-script.sh  # Uses $API_KEY from .bashrc

# Fails in cron
# my-script.sh: API_KEY: unbound variable

Solution:

Source environment in cron job or script:

# Option 1: In cron file
0 */2 * * * root . /etc/environment && /opt/claudeops/bin/claudeops-cron.sh

# Option 2: In script
#!/bin/bash
# Source environment
if [ -f /etc/environment ]; then
    . /etc/environment
fi
if [ -f /opt/claudeops/.env ]; then
    . /opt/claudeops/.env
fi

# Now run commands
claudeops check

Best Practice:

Store production secrets in /etc/environment or dedicated env file:

# /opt/claudeops/.env
ANTHROPIC_API_KEY=sk-ant-xxxxx
DATABASE_URL=postgresql://...

# Make secure
chmod 600 /opt/claudeops/.env
chown root:root /opt/claudeops/.env

Encountered: 2025-10-02 during API key configuration

---

Summary: Key Takeaways

1. Shell Management: Never delete your cwd. Use absolute paths.
2. Cron Jobs: Always include user field in /etc/cron.d/
3. Git Branches: Verify main vs master before pushing
4. PATH: Use absolute paths in cron and sudo contexts
5. Git Ownership: Configure safe.directory for trusted repos
6. Service Dependencies: Properly specify in systemd units
7. Logging: Use structured markdown for human-readable reports
8. Environment: Explicitly source env vars in cron scripts

---

Contributing

If you encounter new issues or lessons during ClaudeOps deployment, please add them to this document!

---

Last updated: 2025-10-02 Lessons learned during production deployment to 65.21.67.254