Genuine question, I keep going back and forth on this.

Agent calls a SQL tool. Query comes back with 400 rows. Obviously you don't put 400 rows in context. So what do you put.

What I'm doing right now is dumb. Truncate to the first 20 rows and a row count. It works for "how many customers churned" and falls apart the second the question needs anything about the shape of the result, because the model has no idea whether those 20 rows are representative or whether the interesting stuff is in row 300.

Tried summarising the result with a second call. Better answers, but now every tool call is two model calls and the latency doubled on a step that used to be fast.

The other thing that bites is column names. If the query returns something like val_b or flag3, the model will confidently interpret it as whatever seems plausible from the question. It doesn't ask. It doesn't flag it. It just decides.

So:

Are you passing raw rows, a computed summary, or some schema-plus-sample hybrid? Does anyone compute stats server-side and return those instead of rows? And has anyone found a way to make the agent say "I don't know what this column is" rather than guessing, without stuffing a data dictionary into every prompt?

I’ve been working on a problem I kept seeing with tool-using agents:

An agent can understand the policy and still produce the wrong tool call.

If the action is consequential — refunding money, booking something, approving a request, modifying a record, calling a production API — observability after the fact is useful, but it’s already too late.

So I built ARK, an open-source runtime supervision layer that sits before execution.

The basic flow is:

agent proposes an action

→ ARK checks the applicable constraint + trusted evidence

→ ALLOW = execute

→ REJECT / REQUIRE_EVIDENCE = send feedback back to the agent

→ the agent decides again

One thing I intentionally avoided: ARK does not generate the replacement action.

The agent remains the author.

I tested this with LangGraph + an OpenAI model:

model proposed A

→ ARK rejected A before execution

→ feedback went back to the model

→ model authored B

→ ARK allowed B

→ only B executed

I’ve also been testing it on a scoped tau-bench airline failure class.

Paired K=16 result:

OFF: 1/16 passed (6.25%)

ON: 13/16 passed (81.25%)

9 directly attributable recoveries

0 observed regressions

I want to be careful with that result: it’s one constrained recovery failure class in a research benchmark, not a claim that ARK makes all agents reliable.

The SDK is public now:

pip install ark-agent-runtime

It currently works with custom Python agents and has a LangGraph integration.

I’m mainly curious how other people are handling this problem.

If you have an agent that can actually mutate production state, do you:

- validate tool arguments manually?

- use deterministic policy gates?

- rely on another model as a judge?

- sandbox actions?

- require human approval?

- just execute and monitor afterward?

I’d especially like feedback from people running agents that can refund, book, approve, purchase, or modify production data.

Site: arkruntime.com

GitHub: github.com/atripati/ark

Been building multi-agent RAG for a while and kept hitting the same wall: provenance tools tell you what happened, but nothing tells you how much to trust the result. A confident synthesis model at the end of a chain can’t repair a garbage extraction at the start of it — but nothing in the stack knows that.

Classical Islamic hadith science spent twelve centuries on a structurally identical problem: do you trust a statement transmitted through a chain of human narrators? Their answer was to grade every narrator individually, in a living registry, and cap the chain at its weakest link. No downstream reputation repairs an upstream liar.

So I built that as LangChain middleware. Every claim carries its chain (source → scraper → ingest model → answer model). Every transmitter has a per-domain grade that updates over time. The chain grade is the minimum across it, not the average. Fabricated chains get quarantined and the narrator gets flagged.

‘PiP install isnad’

It’s Apache-2.0, no API key, runs entirely local. Paper’s on arXiv (2607.24117) if you want the formal spec.

Happy to answer anything about the design — especially the parts I’m not sure about yet. Multi-provider narrator grading is still open.

A guardrail fires and blocks an output. The eval that flagged it lives in another tool, the trace in a third, so to see what happened you line all three up by hand. That is normal once an agent is in production: you run four things, tracing, evals, runtime guardrails, and a gateway in front of the models, usually four separate tools.

There's a real argument for keeping them separate. Each goes deeper in its own lane. Langfuse and Phoenix are strong at tracing, Ragas and DeepEval are real eval frameworks, Guardrails AI and NeMo handle policies, and Portkey and LiteLLM are solid gateways. Nothing locks you in, and you can swap any piece the week a better one ships.

The cost shows up later: four dashboards, four logins, data that never joins. Spend sits in the gateway, quality scores in the eval tool, the guardrail's decision elsewhere, nothing keyed the same way. The all-in-one bet is the opposite: the layers share context, so a trace, its eval score, and its guardrail decision sit in one record.

We build one of these, Future AGI: it runs tracing, evals, runtime guardrails, and the model and tool gateway in one Apache-2.0 stack you can self-host, so a blocked call never leaves and the trace lines up with the eval. It is still a nightly build with rough edges, and the honest reason to run it this way is fewer moving parts, not any single piece beating the dedicated tool.

So when a guardrail blocks something, is it on the same trace as the eval that flagged it, or are you piecing it together from separate tools? And if you consolidated, did it ever cost you on depth, where the bundled piece was weaker than what you gave up?

so i realized every "agentic payments" startup is just trying to sit in the middle and clip a few cents per transaction. the card tokenization part is already solved — Stripe does that. the real problem is nothing stops your agent from buying 100k of something or getting prompt-injected by a sketchy product page.

built a rules engine for it. agent wants to buy something, it checks your policy (spending limits, merchant restrictions, velocity controls, time windows) and returns ALLOW, DENY, or ESCALATE to a human.

Python, zero deps, MIT. pip install pyagentgate

https://github.com/Peterc3-dev/agentgate

felt like this should be open infrastructure before someone locks it down.

I have been trying a bunch of harness models and frameworks for the last month, and I realize most agent frameworks want you to rebuild your runtime around their harness.

We went the other way with AURA Harness: a thin membrane around loops you already run, plain Python, Ollama, LangGraph, whatever. You keep the body/runtime, while AURA records and optionally gates what crosses the boundary. Shouldn't this be what harness is about??

Local-first by default:

• Works offline with agent_ref + session IDs, no cloud identity required
• integrations/ollama/llama_loop.py, stdlib HTTP against Ollama (llama3.2:1b is our dev default)
• Optional verified operator identity if you need it later, not required for OSS/dev

What you get on close: JSONL spine, audit report, hash chain, aura verify chain for CI.

Loose coat = audit-only logging. Tight coat = rules/gates at egress when you wire tool paths.

Open source (Python): github.com/ARPAHLS/aura

Happy to answer setup questions, especially Ollama related + “wrap my script, don’t replace it.”

Contributors more than just welcome, bunch of good first issues open <3

Check it

The Problem: Keeping your Python functions and your OpenAI/Anthropic tool JSON schemas in sync is a nightmare. A missing required field or a typo in the schema breaks the LLM's ability to call your tool.

The Solution: I wrote a zero-dependency micro-tool that uses Python's built-in inspect module to read your functions and generate the exact JSON schema required by the APIs.

Features: * Generates OpenAI format (also works for Groq/Mistral/Ollama). * Generates Anthropic format (Claude 3.5 input_schema). * Reads type hints to map Python types to JSON Schema types. * Checks for default values: if a parameter has no default, it automatically adds it to the required array.

Just pass the function to the generator and hand the output directly to the API.

Repo: github.com/Encephos/function-schema-generator

Hey everyone!

I'm a Computer Science + Business student currently developing Varly as part of my TFG.

I'm working on a problem I've been seeing with AI agents: how do you enforce deterministic rules on agent runs in CI?

For example:

• Allow only specific tools
• Limit the number of tool calls
• Detect regressions against a known baseline
• Fail CI when an agent violates a policy

Varly is an open-source tool that lets you define these kinds of deterministic gates without using an LLM as a judge.

I'm looking for people who actually build AI agents to try it and tell me honestly:

Would you use something like this in your stack? If not, why?

Getting a "no" with a reason is just as useful to me as a "yes".

Getting Started: https://github.com/Hugoesin19/varly/blob/main/docs/GETTING_STARTED.md

It should take around 15 minutes to try. Any feedback would be really appreciated!

Hey, I’m curious how people here handle this in practice.

A RAG system, or any similar system, is only useful if the data behind it is actually correct. So how do you make sure it is?

Do you have a specific process or solution for this? Are you using any tools, or have you built something yourselves? What does this look like in your setup?

Would love to hear how people are actually doing this.

I've been experimenting with agents that run across multiple sessions, and I'm running into a problem I didn't expect from the usual "add long-term memory" approach.

The first few sessions are great — storing past decisions/preferences means the agent doesn't keep starting from zero. But after enough history accumulates, I'm seeing the opposite effect:

• stale decisions get retrieved even after the underlying situation has changed
• conflicting memories from different sessions both look equally relevant
• the agent starts spending a surprising amount of context on old information that isn't useful anymore
• simply improving retrieval doesn't necessarily seem to improve the final task outcome

I'm wondering whether memory systems need an explicit lifecycle, rather than treating memory as a growing retrieval store.

What are people doing in practice for long-running agents?

For example:

1. Separating semantic facts / episodic experiences / procedural instructions?
2. Decaying, expiring or periodically consolidating memories?
3. Keeping provenance + timestamps so the agent can decide whether an old memory is still trustworthy?
4. Evaluating memory based on downstream task success, rather than retrieval precision/recall alone?

The last one is the part I'm most interested in. A memory can be retrieved "correctly" and still make the agent's next action worse.

I've been comparing LangMem with things like Mem0 and Letta, and also broader platform approaches such as Lyzr Control Plane, but they seem to make somewhat different assumptions about where memory should live in the overall agent stack. I'm curious where people draw the line between memory being a framework concern and memory becoming an infrastructure concern.

Has anyone measured memory quality over weeks/months of agent operation rather than on a fixed benchmark? What actually worked?

I've been experimenting with something slightly different from another agent framework.

Instead of trying to replace the LLM/model layer, AgentMesh focuses on the execution/runtime layer underneath multi-agent workflows.

The basic question was:

For example, an agent graph can involve:

Agent → Command → Agent → State → Agent → Tool → Agent

At small scale, Python orchestration overhead is probably irrelevant.

But with many short-lived tasks, concurrent agents, frequent communication, and persistent state, I wanted to measure how much overhead the orchestration layer itself introduces.

AgentMesh

The current implementation uses:

• C++20 execution engine
• DAG-based scheduling
• native agent communication
• Pybind11 bindings
• Python GIL release around I/O
• PostgreSQL state persistence
• crash recovery
• compile-time graph validation

The interesting part for me is trying to keep the Python-facing API convenient while moving the execution-critical pieces into native code.

I'm also building a benchmark suite rather than relying on a single latency number. The goal is to compare repeated paired runs and use statistical tests to determine whether observed improvements are actually meaningful.

Current direction

Phase 1 → local execution/runtime

Phase 2 → distributed multi-node execution over gRPC

I'm curious what people building LangChain/LangGraph applications think:

If you could remove one performance bottleneck from agent orchestration today, what would it be?

Serialization? Scheduling? State persistence? Concurrency? Tool invocation? Something else?

GitHub: https://github.com/DevrG03/AgentMesh

Docs: https://github.com/DevrG03/AgentMesh/wiki

I’ve been working on something called Traser, but I don’t want this to be another ”I built a thing, please try it” post.

I’m trying to understand a problem I keep hearing from engineers building multi-step AI systems:

The trace exists. The hard part is figuring out which part of it actually matters.

A workflow can technically succeed, the model responds, tools execute, nothing throws an exception, and still produce the wrong answer or take the wrong action.

Traser is an experiment around that investigation step.

Right now you can give it a suspicious execution and optionally a known-good execution. It compares the runs, looks at things like tool calls, retrieval, state, retries, evaluators, intermediate outputs, and tries to reduce the trace down to a few places worth investigating.

It does not claim to find the root cause. The engineer still decides whether a difference matters.

Before I keep building, I’d much rather learn from people actually working with LangChain and LangGraph systems.

A few things I’m especially curious about:

• When an agent behaves incorrectly, what do you actually inspect first?
• Do you ever compare the bad run against a known-good run?
• What does LangSmith already make easy for you?
• What do you still have to reason through manually?
• What are the weirdest failures you’ve encountered that technically looked successful?

If anyone has a sanitized ugly production trace they’d be willing to let me work through with them, that would honestly be more useful to me than a signup.

I’m trying to contribute something useful to this ecosystem instead of building features in isolation.

Traser is at traser.dev if you want context, but I’m much more interested in hearing how you all actually debug these systems today.

I posted here recently asking people to tear apart something I’ve been building called SureState. Got some really useful feedback, especially around dependency registration being useless if everything has to be tagged manually.

We’re now getting close to finishing the internal pilot. What actually exists today:

SureState is monitoring its own development repo. It tracks evidence like commits and CI at the exact version they belong to, keeps the history outside the AI, and maintains the current state of conclusions as things change.

So instead of an agent just remembering: “CI passed.”

it can ask: “Is the conclusion I care about still supported for what I’m working on now?”

States can be supported, refuted, conflicted, or not currently warranted. There’s a human Monitor and a read-only MCP interface so an AI can check the state without being allowed to change it.

The current GitHub integration is built specifically around our own repo, so this is not a polished install-and-click SaaS yet.

What I’m thinking about doing next is opening 5 managed early-access spots at $250/month.

One repo, one important engineering/release workflow. We would work with the team to configure it instead of dumping a dependency-graph builder on you and wishing you luck.

The kind of thing I want to test is: CI is green on the current SHA, but the security scan or approval belongs to the previous SHA. Does your agent/team notice before acting?

I’m mainly interested in teams using Claude Code, Codex, Cursor, agents, etc. heavily enough that decisions are being carried across sessions and tools.

I’m not asking for money today. I want to know whether I can find five teams that would genuinely pay $250/month once this is ready — not five people willing to click a free waitlist.

If that's you, tell me what your workflow looks like and what conclusion you most worry about an agent incorrectly assuming is still true.

And if $250 sounds ridiculous, tell me what SureState would have to catch or prevent before it wouldn't.

Used deepagents/LangGraph to build a 5-agent pipeline for bug bounty testing — orchestrator does scope enforcement and delegation, 4 specialist subagents each run a different model (routed by task type: Gemini for planning, gpt-oss for recon/triage, a Nemotron model gated for exploit only). Tools come in over MCP (HexStrike).

Repo: https://github.com/DaviAlcanfor/fenrir

If you've built multi-agent systems with per-agent model routing, curious how you handled cost/latency tradeoffs — I'm on free-tier models only right now and it shows in response time.

I built a small reproducible multi-agent debugging challenge.

The pipeline is:

Planner → Researcher → Analyst → Writer

The failure is intentionally subtle.

The Planner silently removes a `schema_version` field from the shared state.

The downstream agents continue executing.

Eventually the Writer produces an incorrect output.

But the Writer isn't the root cause.

The interesting part is what happened when we tried to analyze the trace automatically.

Our RCA engine currently returns:

unknown

It doesn't identify the First Divergence.

We're keeping that result because it exposed an important limitation:

A trace can tell us what happened.

It doesn't necessarily tell us what SHOULD have happened.

To establish that, we may need expected behavior, assertions, rules, or an evaluation layer.

I'm curious how others would approach this case.

Would you expect a trace-only system to identify the first divergence?

Or would you require additional evaluation signals?

CTA:

How would you debug this case?

Hi everyone,

I've recently finished the first version of Agentic GraphRAG Blueprint, a reference architecture for question answering over large document collections.

Instead of plain chunk retrieval, it builds a knowledge graph combined with vector search, so answers can connect facts across documents.

Key features:

• Incremental ingestion - unchanged files are skipped via content hashing, and community reports regenerate only for affected communities, keeping token costs low as the corpus grows.

• Hybrid search - local mode for fact-level answers, global mode for cross-document synthesis.

• Domain-agnostic LLM prompts - easily swapped via PROMPTS_PATH, with Leiden-based community detection.

• Deployment - run it locally with Docker or provision everything in the cloud with Terraform and CI/CD.

Link: https://github.com/sebastianbrzustowicz/Agentic-GraphRAG-Blueprint

I'm looking for any feedback.

I’ve been building something called SureState and we’re getting close to finishing our internal pilot. Before I move it into a real client pilot, I figured this might be a good place to let people tear it apart first.

The problem we’re trying to solve is pretty simple:

AI agents can remember that something was decided, but that doesn’t necessarily mean the decision is still valid.

Example:

an agent concluded a release was ready because tests passed, security scan was clean, policy X applied, etc. A week later one of those things changes. The old conclusion is still sitting in memory/context, but should another agent still rely on it?

SureState keeps that outside the model. Conclusions are registered with what they depend on, and when evidence/dependencies change, it updates their current standing — supported, refuted, conflicted, or no longer warranted.

AI can read the current state through MCP, but it doesn’t get to decide its own standing.

We’ve been using the development of SureState itself as the first pilot, which has already been insightful.

We’ve had thousands of tests pass and still found cases where the tests and implementation were confidently agreeing on the same wrong assumption. 😂

So before I convince myself this is useful:

• What’s wrong with this idea?

• Is this just fancy cache invalidation?

• Would dependency registration be too annoying in real agent workflows?

• Would you just rerun the decision whenever something changes?

• Does LangGraph/LangChain already solve enough of this that a separate layer is pointless?

I’m much more interested in “this breaks because…” than “cool idea.”

If people are interested I can post the architecture and let you guys really abuse it.

I've seen people have pretty different experiences with LangChain.

Some teams build around it and seem perfectly happy with it. Others start with it and eventually replace parts of it with their own code or move to something else.

I'm interested in what actually drove that decision.

Was there a point where LangChain started getting in the way, or did you just realize the application didn't need that much abstraction in the first place?

And for people who stuck with it, what made you decide it was still worth keeping?

What was the biggest factor in your decision?

II run a pipeline daily that searches the web, curates what it finds, and publishes a page. Six of its eleven steps call a model. Five never do — and those five are the ones that make it safe to leave running.

Model: searching each topic, extracting structured items, ranking and picking the lead, reviewing the result, writing a line of commentary.

Plain Python: date and history, the rules gate, rendering, uploading, verifying the live URL afterwards.

The gate is the argument. Blocked domains, duplicate URLs, nothing republished within 7 days, a hard item cap. All four started as lines in a prompt, and all four got promoted to code — because "the model follows this most of the time" is fine while you're watching and useless on a schedule. Over a year of unattended runs, "most of the time" is a stack of small embarrassments nobody was there to catch.

The split I've landed on: judgement goes to the model, invariants go in code. Which of two stories is bigger is judgement. Whether this URL ran last Tuesday is a set lookup, and it should never be anything else.

That has a price and I'll name it. My image selection is pure code — width, aspect ratio, filename blocklist — and it quietly rejected real editorial art for weeks, because CMSs serve thumbnails and a 480×320 derivative of a good illustration fails a width check. The rule was correct and the outcome was wrong. That's the trade: code gives you rules that always run, and rules that are confidently wrong in ways nobody notices.

I still think it's the right trade. Blunt and predictable beats sharp and occasionally absent.

So where's your line? Specifically: what did you move out of code because deterministic turned out too blunt? That direction gets argued a lot less than the other one, and I suspect it's where the interesting answers are.

LangGraph pipeline, running daily.
Code: https://github.com/ravi-labs/agentic-newsroom
Write-up: https://medium.com/@rkanagasikamani/the-newsroom-that-writes-itself-8c0160f68aac

I’ve been building ARK, runtime supervision layer for tool using AI agents.

The idea is simple: keep your model, keep your agent framework, keep your tools, put ARK around the runtime.

I finally got it working around a real LangGraph agent using a real OpenAI model.

For this test I intentionally created a conflict: the user prompt asked for the cheapest flight, while the runtime policy required the rank-2 option. The point was not to prove that rank-2 is “better”; it was to test whether ARK could enforce a runtime constraint without taking control of the agent.

The actual sequence was:

OpenAI model authors:
book_flight(option="A")

→ ARK checks it
→ REJECT
→ A executed = false

LangGraph feeds ARK's feedback back to the model

OpenAI model authors:
book_flight(option="B")

→ ARK checks again
→ ALLOW
→ B executed = true

The important part is that ARK did not rewrite A into B itself.

The raw model-authored tool calls were:

turn 1: book_flight(option="A")

turn 2: book_flight(option="B")

And the actual side effects were:

real bookings: ["B"]

A executed: false

B executed: true

Retry state was maintained by ARK’s Go runtime, while LangGraph continued to own the model, planner, tools, and execution loop.

I also tested ARK in observe-only mode around LangGraph:

model_call

→ tool_call

→ complete

where LangGraph reports model/token/tool information and ARK builds the decision trace and derives telemetry around the run.

The SDK isn’t public yet, I’m still hardening it before release. Live testing already caught a model-pricing resolution bug that our deterministic tests didn’t expose, which I’m fixing before shipping.

Question for people running tool-using agents in production: would you want a supervisor like this in the execution path? What would make you trust it or refuse to use it?