Provider-agnostic multi-LLM deliberation. Three phases — independent responses, cross-model anonymized ranking, chairman synthesis. Credentials come from explicit plugin configuration or pro-workflow-specific CLI variables. Persists transcript to a wiki page when --wiki <slug> is passed. Use when the user wants multiple AI perspectives, consensus-building, or the "LLM Council" approach for high-stakes reviews, plan critique, or contested learning rules.
git clone https://github.com/rohitg00/pro-workflow.git--- name: llm-council description: Provider-agnostic multi-LLM deliberation. Three phases — independent responses, cross-model anonymized ranking, chairman synthesis. Credentials come from explicit plugin configuration or pro-workflow-specific CLI variables. Persists transcript to a wiki page when --wiki <slug> is passed. Use when the user wants multiple AI perspectives, consensus-building, or the "LLM Council" approach for high-stakes reviews, plan critique, or contested learning rules. user-invocable: true --- # LLM Council Karpathy's LLM Council pattern with Anthropic and OpenAI-compatible providers. ## When to use - High-stakes plan review (`/plan` crosses N-file threshold) - Conflicting learning-rules → re-resolve via vote - User invokes `/council "<query>"` or `/wiki council` - Architecture decisions where you want multiple viewpoints captured - Persisting deliberation as a wiki page for future reference ## Three phases 1. **Independent**: each model answers in parallel 2. **Ranking**: each model ranks anonymized peer responses 3. **Synthesis**: chairman model reads all responses + rankings → final answer ## Provider config Configure optional keys in the plugin configuration dialog. Use the `providers` MCP server's `run_provider_task` tool with `task: "council"` and `args: ["run", "<query>", "--provider", "openai"]`. For provider status, pass `args: ["providers"]`. Never request keys in chat or read them from existing credentials or shell configuration. For standalone CLI use, explicitly set a pro-workflow-specific variable below. The first configured provider is the default; `--provider` selects one explicitly. See [provider configuration](../../references/provider-configuration.md). | Standalone CLI variable | Provider | Default base URL | |---------|----------|------------------| | `PRO_WORKFLOW_ANTHROPIC_API_KEY` | Anthropic | `https://api.anthropic.com` | | `PRO_WORKFLOW_OPENAI_API_KEY` | OpenAI | `https://api.openai.com/v1` | | `PRO_WORKFLOW_OPENROUTER_API_KEY` | OpenRouter | `https://openrouter.ai/api/v1` | | `PRO_WORKFLOW_FIREWORKS_API_KEY` | Fireworks | `https://api.fireworks.ai/inference/v1` | | `LLM_COUNCIL_BASE_URL` + `PRO_WORKFLOW_LLM_COUNCIL_API_KEY` | Custom OpenAI-compat | (user-supplied) | Override per-run with `--provider openai|anthropic|openrouter|fireworks|custom`. Default model rosters per provider live in `scripts/council.js` and can be overridden via `--models` CSV and `--chairman <id>`. ## Commands In a plugin session, pass these runner arguments through `run_provider_task`. The direct commands below are for standalone CLI installations with explicit credentials. ``` node $SKILL_ROOT/scripts/council.js run "<query>" [--models id1,id2,id3] [--chairman id] [--provider <name>] [--wiki <slug>] node $SKILL_ROOT/scripts/council.js providers node $SKILL_ROOT/scripts/council.js show <session-id> ``` `--wiki <slug>` writes the full transcript to `<wiki>/derived/council/<session-id>.md` and registers it via `wiki-cli.js page` so it shows in FTS5 search. ## Output Each session writes: ``` ~/.pro-workflow/council/<session-id>/ ├── config.json # query, models, chairman, provider ├── phase1_responses.json # raw API responses per model ├── phase2_rankings.json # anonymized ranking outputs ├── phase3_synthesis.txt # chairman's final answer └── final_output.md # human-readable bundle ``` Console prints the markdown bundle. Pipe to `pbcopy` / `tee` as needed. ## Hard rules 1. Never skip the ranking phase. It's the core of the council pattern. 2. Save raw responses to disk verbatim. No summarization in storage. 3. Anonymize responses for ranking — models see `Response A/B/C/...`, not peer names. 4. The chairman sees both real names AND rankings. 5. Display all three phases to the user. No phase elision. ## Cost awareness The script logs per-call latency + tokens on supported providers. Multiply by your provider rate to estimate. Council cost grows linearly with `len(models)^2` (each model ranks all others) plus the chairman. Default council size: 3-5 models. More models = exponentially more ranking calls. ## Use with wiki ``` /wiki council agent-memory "should we adopt episodic memory in our agents?" ``` Loads `agent-memory` wiki context as system prompt prefix, runs council, persists transcript as `wiki/derived/council/<id>.md`. The transcript becomes searchable via `/wiki ask`.
[{"step":"Configure the LLM Council plugin with your preferred models. For CLI users, set environment variables: `LLM_COUNCIL_MODELS=\"gpt-4o,claude-3-5-sonnet,llama-3.1-405b\"`. For plugin users, edit the configuration file to specify model endpoints and API keys.","tip":"Use models with different strengths (e.g., one for cost analysis, one for technical feasibility, one for risk assessment) to ensure diverse perspectives."},{"step":"Run the council with your prompt template, including placeholders for [TOPIC/PROBLEM], [MODEL_A/B/C], [SPECIFIC_FOCUS_AREA], and optionally [WIKI_SLUG] for transcript persistence.","tip":"For high-stakes decisions, include a 'confidence threshold' in your prompt (e.g., 'Only consider solutions with >80% confidence from all models')."},{"step":"Review the anonymized rankings and synthesis. Use the 'chairman's report' as a decision document or to identify gaps in the council's reasoning.","tip":"If models disagree sharply, add a 'tiebreaker' phase where they must address each other's weaknesses directly."},{"step":"Implement the recommendations and track outcomes against the council's projections. Use the wiki transcript for audit trails or to refine future council sessions.","tip":"Schedule a follow-up council session after 30 days to validate results and adjust strategies based on real-world data."}]
No install command available. Check the GitHub repository for manual installation instructions.
git clone https://github.com/rohitg00/pro-workflow/tree/main/skills/llm-councilCopy the install command above and run it in your terminal.
Launch Claude Code, Cursor, or your preferred AI coding agent.
Use the prompt template or examples below to test the skill.
Adapt the skill to your specific use case and workflow.
Convene the LLM Council to analyze [TOPIC/PROBLEM]. Use the following steps: 1) Generate independent responses from [MODEL_A], [MODEL_B], and [MODEL_C] (do not reveal identities). 2) Cross-model anonymized ranking of the top 3 strongest arguments or solutions. 3) Chairman synthesis to produce a final consensus report. Persist the transcript to a wiki page with slug [WIKI_SLUG] if needed. Focus on [SPECIFIC_FOCUS_AREA] and avoid assumptions outside provided context.
### LLM Council Report: Optimizing Cloud Costs for Acme Corp (Q3 2024) **Phase 1: Independent Responses** *Model A (GPT-4o)*: Recommended migrating 60% of workloads to spot instances with auto-scaling policies, projecting 28% cost reduction but warned of 12% uptime risk. Suggested using AWS Cost Explorer for 30-day baseline analysis. *Model B (Claude 3.5 Sonnet)*: Proposed a tiered approach: 40% reserved instances for steady-state workloads (15% savings), 30% spot instances for batch jobs (22% savings), and 30% on-demand for critical services. Emphasized implementing AWS Compute Optimizer for right-sizing. *Model C (Llama 3.1 405B)*: Advocated for a FinOps-driven approach with tagging enforcement, budget alerts at 80% threshold, and automated shutdown policies for dev environments. Estimated 24% savings with 99.9% uptime reliability. **Phase 2: Cross-Model Anonymized Ranking** 1. **Tiered Spot/Reserved/On-Demand Hybrid** (Model B) - 9.2/10 - Strengths: Balanced risk/reward, actionable migration path, leverages multiple pricing models - Weaknesses: Requires more upfront configuration 2. **FinOps Automation with Tagging** (Model C) - 8.7/10 - Strengths: Cultural shift toward cost ownership, scalable enforcement - Weaknesses: Lower immediate savings, depends on organizational maturity 3. **Aggressive Spot Migration** (Model A) - 7.5/10 - Strengths: Highest theoretical savings, simple to implement - Weaknesses: High operational risk, requires robust monitoring **Phase 3: Chairman Synthesis** The council recommends **Model B's tiered approach** as the optimal strategy for Acme Corp, with the following modifications: 1. **Immediate Actions (Week 1-2)**: - Implement AWS Compute Optimizer for workload analysis - Set up budget alerts in AWS Cost Management at 75% threshold - Enforce mandatory tagging with 'cost-center' and 'environment' keys 2. **Migration Phase (Week 3-6)**: - Reserve 35% of steady-state workloads (target: m5.large instances) - Migrate 30% of batch jobs to spot instances with checkpointing - Maintain 35% on-demand capacity for critical services 3. **Governance Layer**: - Establish a monthly FinOps review with engineering leads - Create a 'cost incident' Slack channel for real-time alerts - Document rollback procedures for spot instance interruptions **Projected Outcomes**: - Q3 Savings: 21-24% (vs. current baseline) - Uptime Impact: <1% (within SLA) - Implementation Effort: Medium (2 FTEs for 4 weeks) *Transcript saved to wiki: https://acme-corp.wiki/llm-council-cloud-costs-q3-2024*
skills-collection
Take a free 3-minute scan and get personalized AI skill recommendations.
Take free scan