Established the initial goals, parameters, and constraints for a local agent.
Local AI Agent
Homegrown Intelligence
An experiment in building a private, locally run AI agent around my own tools, data, and workflows — designed around how I actually work.
What am I trying to build?
Starting parameters for a local personal agent that keeps my data under my control.
Keep local and cloud models separate. Use cloud models when more advanced reasoning is necessary, but I will act as the bridge deciding what context crosses between them.
Don’t try to rebuild advanced chatbots. Prioritize useful access, security, tools and integration over raw model power.
Build for a real personal computer. Keep the project viable on hardware I can realistically own and operate at home.
Models are components of the system. Keep my data, memory, and core tools separate from the model so models can be replaced without rebuilding the agent.
Solve real problems. Build capabilities around whatever gets in my way, rather than what's possible simply because the technology exists.
Automate without outsourcing judgment. Let the agent notice, suggest, and take initiative, but leave meaningful decisions to me.
What should it do?
Capabilities I want to build toward, organized by the role I want them to play.
Knowledge
Keep track of what I know and help me find it again. Work across Obsidian, local files and memory to retrieve context, connect ideas and maintain a useful knowledge base.
Assistance
Help me work with information once I have it. Research, summarize, compare and pull together material from different sources.
Awareness
Understand what I'm doing when I choose to give it that context. Use on-demand computer vision and screen context to answer questions and eventually observe workflows.
Automation
Handle work that doesn't need my constant involvement. Run scheduled searches, cross-reference email and calendar, send reminders and carry out approved routine actions.
Limitations & proposed workarounds.
Practical challenges and how I might address them.
Hardware. My local model will have to share limited RAM and VRAM with everything else I use my computer for. I'll prioritize efficient models and tools before upgrading hardware.
Availability. If my computer is sleeping, my local agent won't be accessible. I'm going to accept that limitation rather than maintaining a permanent server, and revisit if it becomes inconvenient.
Expertise. As this experiment extends outside the scope of my formal training, I'll be relying on AI assistance for development. To reduce the risks that introduces, I'll use established protocols and primary documentation for safety and data security, and seek outside expertise where warranted.
Maintenance. Since I own the system, I'll have to upkeep it. I intend to keep the architecture simple and modular so that maintaining my local agent doesn't cost more time than it saves.
The #1 priority is security.
Credentials
Keeping secrets separate.
Passwords, API keys and other credentials shouldn't be stored somewhere the model can read them. I'll use established methods for authentication and let the model interact with authenticated services through controlled tools.
Permissions
Limiting what it can do.
I'll start by giving my agent as little access as possible and add permissions as needed. More consequential actions, especially deletion or anything public-facing, will require my approval.
Privacy
What leaves my computer.
Accessing the internet will sometimes require sending information elsewhere. I'll limit what gets shared to what's actually needed for the task.
Trust
Defaulting to distrust.
The model will sometimes misunderstand, hallucinate, or follow instructions it shouldn't. Important security rules should be enforced by code rather than left to the model's judgment.
Project Log