LangChain shows AI agents aren’t human-level yet because they’re overwhelmed by tools - Related to storage, tools, shows, agents, human-level

LangChain shows AI agents aren’t human-level yet because they’re overwhelmed by tools

With AI agents showing promise, organizations have to grapple with figuring out if a single agent is enough, or if they should invest in building out a wider multi-agent network that touches more points in their organization.

Orchestration framework enterprise LangChain sought to get closer to an answer to this question. It subjected an AI agent to several experiments that found single agents do have a limit of context and tools before their performance begins to degrade. These experiments could lead to a improved understanding of the architecture needed to maintain agents and multi-agent systems.

In a blog post, LangChain detailed a set of experiments it performed with a single ReAct agent and benchmarked its performance. The main question LangChain hoped to answer was, “At what point does a single ReAct agent become overloaded with instructions and tools, and subsequently sees performance drop?”.

LangChain chose to use the ReAct agent framework because it is “one of the most basic agentic architectures.”.

While benchmarking agentic performance can often lead to misleading results, LangChain chose to limit the test to two easily quantifiable tasks of an agent: answering questions and scheduling meetings.

“There are many existing benchmarks for tool-use and tool-calling, but for the purposes of this experiment, we wanted to evaluate a practical agent that we actually use,” LangChain wrote. “This agent is our internal email assistant, which is responsible for two main domains of work — responding to and scheduling meeting requests and supporting consumers with their questions.”.

LangChain mainly used pre-built ReAct agents through its LangGraph platform. These agents featured tool-calling large language models (LLMs) that became part of the benchmark test. These LLMs included Anthropic’s Claude [website] Sonnet, Meta’s [website] and a trio of models from OpenAI, GPT-4o, o1 and o3-mini.

The organization broke testing down to enhanced assess the performance of email assistant on the two tasks, creating a list of steps for it to follow. It began with the email assistant’s customer support capabilities, which look at how the agent accepts an email from a client and responds with an answer.

LangChain first evaluated the tool calling trajectory, or the tools an agent taps. If the agent followed the correct order, it passed the test. Next, researchers asked the assistant to respond to an email and used an LLM to judge its performance.

For the second work domain, calendar scheduling, LangChain focused on the agent’s ability to follow instructions.

“In other words, the agent needs to remember specific instructions provided, such as exactly when it should schedule meetings with different parties,” the researchers wrote.

Once they defined parameters, LangChain set to stress out and overwhelm the email assistant agent.

It set 30 tasks each for calendar scheduling and customer support. These were run three times (for a total of 90 runs). The researchers created a calendar scheduling agent and a customer support agent to enhanced evaluate the tasks.

“The calendar scheduling agent only has access to the calendar scheduling domain, and the customer support agent only has access to the customer support domain,” LangChain explained.

The researchers then added more domain tasks and tools to the agents to increase the number of responsibilities. These could range from human resources, to technical quality assurance, to legal and compliance and a host of other areas.

After running the evaluations, LangChain found that single agents would often get too overwhelmed when told to do too many things. They began forgetting to call tools or were unable to respond to tasks when given more instructions and contexts.

LangChain found that calendar scheduling agents using GPT-4o “performed worse than [website], o1 and o3 across the various context sizes, and performance dropped off more sharply than the other models when larger context was provided.” The performance of GPT-4o calendar schedulers fell to 2% when the domains increased to at least seven.

Other models didn’t fare much enhanced. [website] forgot to call the send_email tool, “so it failed every test case.”.

Only [website], o1 and o3-mini all remembered to call the tool, but [website] performed worse than the two other OpenAI models. However, o3-mini’s performance degrades once irrelevant domains are added to the scheduling instructions.

The customer support agent can call on more tools, but for this test, LangChain showcased [website] performed just as well as o3-mini and o1. It also presented a shallower performance drop when more domains were added. When the context window extends, however, the Claude model performs worse.

GPT-4o also performed the worst among the models tested.

“We saw that as more context was provided, instruction following became worse. Some of our tasks were designed to follow niche specific instructions ([website], do not perform a certain action for EU-based end-consumers),” LangChain noted. “We found that these instructions would be successfully followed by agents with fewer domains, but as the number of domains increased, these instructions were more often forgotten, and the tasks subsequently failed.”.

The organization introduced it is exploring how to evaluate multi-agent architectures using the same domain overloading method.

LangChain is already invested in the performance of agents, as it introduced the concept of “ambient agents,” or agents that run in the background and are triggered by specific events. These experiments could make it easier to figure out how best to ensure agentic performance.

Thunderful Gothenburg showcased today that it has rebranded with a new name: Stormteller Games. , this new name signifies a new......

Delhivery’s consolidated net profit zoomed 114% to INR [website] Cr in Q3 FY25 from INR [website] Cr in the year-ago quarter on the back of strong growth in i......

Swedish startup Agteria Biotech raises €6M to reduce methane emissions from cattle

Agteria Biotech, a Swedish startup dedicated to a 1 per cent reduction in global greenhouse gas emissions by reducing methane emissions from cows burping and farting, has raised €6 million in Seed funding.

Methane — 28 times more potent than carbon dioxide and responsible for 30 per cent of the current rise in global temperatures — has become a critical climate priority.

Methane emissions from cattle accounts for 5 per cent of global greenhouse gas emissions.

“Our goal from the start has been to create a solution that is not only effective but also scalable, affordable, and, above all, safe,” unveiled Martin Blomberg, CEO of Agteria.

“Methane levels in the atmosphere have never been higher, in 2024 global temperatures surpassed [website] degrees for the first time, and reducing methane is the most impactful short-term action we can take to combat warming.”.

Founded in 2023, Agteria has developed a patent-pending molecule that significantly reduces methane emissions from cattle. The solution is scalable, low-cost, and offers the highest methane reduction per euro.

The firm’s solution has already demonstrated high efficacy in several in-vivo trials, and has attracted strong interest from both beef and dairy companies as well as animal feed companies.

Industrifonden and AgriZeroNZ led the round, with continued support from existing investors Norrsken Launcher and Mudcake.

“Agteria’s progress in a short time is remarkable, driven by a small, highly ambitious, and energetic team. We’re excited to support their journey as they bring a unique, scalable, science-backed innovation with the potential to significantly reduce agricultural greenhouse gas emission.” revealed Mala Valroy, Investment Manager at Industrifonden.

“We’re actively scanning the world for innovative ventures to invest in to help farmers reduce emissions. We’re proud to back Agteria to accelerate the development of a practical, effective solution suitable for grazing systems in New Zealand and around the world,” revealed Wayne McNee, Chief Executive at AgriZeroNZ.

The new funding will enable the firm to advance toward regulatory approval for its novel methane-reducing products for cattle, focusing on essential safety studies for animals, consumers, and the environment.

It allows its consumers to automate cloud storage provisioning and reduces human intervention in the entire work-cycle.

RBI also underlined that the bank has undergone an external audit to validate these compliances.

RBI noted that the bank has taken measures to address......

Table of Contents Table of Contents How WhatsApp can change the game? This can’t come sooner.

WhatApp might soon automatically detect and translate yo......

Lucidity Bags $21 Mn To Power Cloud Storage Solutions For Enterprises

It allows its consumers to automate cloud storage provisioning and reduces human intervention in the entire work-cycle.

Founded in 2021 by Vatsal Rastogi and Nitin Bhadauria, Lucidity offers AI-enabled software for cloud storage which allows its clients to directly manage their cloud infrastructure in an economical and efficient manner.

The New York and Bengaluru-based startup plans to use the fresh capital to expand its go-to-market team and bolster solutions for storage management problems for the enterprises.

Cloud storage management startup Lucidity has secured $21 Mn (around INR 182 Cr) in a Series A funding round led by WestBridge Capital, along with participation from existing investor Alpha Wave.

The NewYork and Bengaluru-based startup plans to use the fresh capital to expand its go-to-market team and bolster solutions for storage management problems for the enterprises.

Founded in 2021 by Vatsal Rastogi and Nitin Bhadauria, Lucidity offers AI-enabled software for cloud storage which allows its clients to directly manage their cloud infrastructure in an economical and efficient manner.

It allows its consumers to automate cloud storage provisioning and reduces human intervention in the entire work-cycle.

“Lucidity delivers the only platform for ITOps and DevOps organisations to automatically manage and optimize their block storage in real- time across all three major cloud providers while significantly reducing costs. As a result, we’re honored by the ongoing interest we’ve received and the opportunity to work with some of the largest enterprises in the world to empower them to seamlessly manage their cloud storage for the first time,” noted Bhadauria.

Since its inception, Lucidity asserts to have achieved 400% year-over-year growth, driven by industry-first innovations that automate and optimize cloud block storage utilisation.

The startup says that its NoOps (no operations), autonomous and application-agnostic layer seamlessly integrates with existing applications and environments without requiring a single line of code to be changed.

Compatible with AWS, Azure, and Google Cloud, Lucidity’s platform automates storage optimization, helping IT and DevOps teams of enterprises to slash costs, prevent downtime, and streamline operations, the statement added.

In 2022, Lucidity raised $[website] Mn in a seed funding round led by AlphaWave Investments.

The firm has raised a total of $31 Mn in funding to date and counts the likes of BEENEXT, Blume Ventures and Boldcap among its investors.

As per Inc42, homegrown startups cumulatively netted more than $12 Bn in fresh funds in 2024, up over 20% from the $10 Bn raised in 2023.

Microsoft has released the KB5051974 cumulative enhancement for versions 22H2 and 21H2, adding security fixes and patching a memory leak. However, as Bleep......