AI agents creation and working, with Autogen, LM Studio
Comprehensive Guide to Building AI Agents with Autogen and LM Studio
Introduction
In the rapidly advancing field of artificial intelligence (AI), building AI agents that can communicate and interact like humans is at the forefront of technological innovation. Today, we dive into the intricacies of AI agents’ creation and functioning, utilizing powerful tools like LM Studio and Autogen. This extensive guide will walk you through the setup and operation of these tools, effectively enabling AI agents to communicate, carry out tasks, and even engage in group chats.
Setting Up Your Environment
Essential Tools: LM Studio and Visual Studio Code
To begin with AI agent creation, the first step involves setting up a local server with LM Studio, which is instrumental in downloading and deploying the language model of your choice. Once LM Studio is configured, it interacts with the AI model locally, mitigating the need for constant internet connectivity and allowing faster data processing.
Visual Studio Code (VS Code) plays a pivotal role here as your editor for running Python code which controls the interactions with your AI model.
Integrating Autogen
Autogen, developed by Microsoft, is a transformative tool that simplifies the process of building conversational programs between different AI agents. It plays a critical role in defining how each AI agent behaves and communicates.
Creating AI Agents with Autogen
Configuration for Individual and Group Chats
You can start by setting multiple models in LM Studio, each potentially linked to a different agent. However, a standard setup allows all agents to use a single model unless specific scenarios require otherwise. The process begins by importing the necessary classes from Autogen and assigning each agent a distinct role or function.
For group chats, which involve multiple AI agents simultaneously interacting, further configurations might be necessary to ensure smooth communication without conflicts. Group chat settings allow agents to share information and come up with collaborative responses, closely mimicking a human team’s discussion dynamics.
Implementing Conversations
Simple Dialogues
In a basic setup where two agents communicate (e.g., exchanging jokes or simple information), the implementation involves setting up a conversation initiator and responder in the code. Here, the interaction is relatively straightforward, typically involving question and answer or an exchange of ideas.
Advanced Scenarios: Financial Analysis
A more complex example involves setting up agents that specialize in various aspects of financial analysis. Different agents can be configured to analyze data, review news, and summarize insights. This multi-agent setup can simulate a financial team where each member contributes to the final investment advice based on varied data points.
Testing and Refining the Setup
Running and Analyzing Test Conversations
Once the agents are configured, the next step involves running test conversations to observe and tune the interactions. This involves checking the logical flow of conversations, the relevancy and accuracy of responses, and ensuring that each agent sticks to its defined role.
Handling Errors and Improvements
The iterative process of testing often reveals bugs or inefficiencies that need refining. This could involve adjusting the conversation max turns, refining the agents’ roles, or even changing parameters in the Autogen or LM Studio settings to optimize performance and response accuracy.
Conclusion: Towards Intelligent Conversational Agents
Creating intelligent AI agents capable of realistic and valuable interactions involves combining innovative tools like Autogen and LM Studio with thoughtful configuration and diligent testing. While the initial setup may involve overcoming challenges concerning configurations and refining agent behaviors, the end result is a robust framework where AI agents can autonomously handle tasks and communicate effectively.
For those interested in delving into AI agent creation, remember that continuous learning and adaptation are key. Experiment with different setups, engage with the community for insights, and keep refining your models to achieve better accuracy and reliability in AI-powered conversations.
For detailed step-by-step code examples, further reading, and community discussions, feel free to explore more on our channel or in dedicated forums online. Embrace the journey of AI innovation and contribute to shaping the future of smart agent interaction!
[h3]Watch this video for the full details:[/h3]
AI agents creation and working, with Autogen, LM Studio
All codes and models are run locally.
https://buymeacoffee.com/learnbydoing
[h3]Transcript[/h3]
everyone welcome to my channel again today I’m going to present you the project that I have successfully run uh in my local environment to build the AI agents and let them talk with each other um first of all uh the tools I’m going to use first of all is LM studio uh so with LM Studio you can download the model that you want to test and also to load it and to start a server so to let the vs code module to actually access to the local server and interact with the large language model that you prefer and then of course we need the uh Visual Studio visual studio uh software installed so that you can run the python code very easily and the last and most important thing is that the autogen autogen is a package and also the um I think a package is a r name because I’m going to Import in the python environment uh this autogen is uh I think developed by Microsoft and is very useful for me to um actually build this uh conversation um conversation program uh between different agents uh so you need to pay attention to two parts the first part is LM Studio page it will tell you how autogen can interact with LM Studio it is very in detail you can test all the code of course the second part is about group chat group chat is actually uh uh more advanced so if you check this page of the uh LM Studio it gave the example of two agents chats so is uh just two agents com uh talking back and forth but if you are talking about uh the group chat actually the group chat will invite multiple agents to interact with each other okay so I invite you to check these documents it will be very useful for you to test a code so uh let’s come to the code part uh from the basic I just mimic what the document says so first of all uh I uh set up to models okay so uh if you uh set for example this Gemma refers to this Gemma model and also the URL key is uh uh the local LM Studio you can find it and also the API key is LM Studio okay so basically setting up the different model will be useful when the agents will link to these models uh and you can see here the second part is to from autogen import conversible agents uh this part is actually a defining the different agents and which model it will it will use so for me I always use only one model but if you want one agent to use one model the other use another one it is uh uh I think is possible but you need more configurations of LM studio and so on okay so basically as this moment I in my LM Studio I chose the MST model uh but here I leave it like this it kind wrong because I think LM studio will recognize uh this kind of Link um directly uh without without actually checking what what which model you are referring to because actually the the server is working with one model so when you link to the server it will work okay so um this is uh uh the code that you need to run the conversation it will produce the chat results okay so who will start the conversation Jack will start the conversation and it will talk with Emma and the starting mage m is Emma tell me a joke Max terms will Define uh maximum of the of the conversation uh passage uh is like uh if the maximum is two actually you have a you see here you have a one two three four uh I think uh terms one term two terms one term is uh one back and forth uh two terms is in total uh two two back and forth yeah so you can see here the result is uh very good uh Emma tell me a joke and Emma starts to tell tell a joke okay then Jack says uh that was a great joke I’m glad it brought a smile to your face as well uh then Emma absolutely agree so they’re actually doing this conversation right you can check and the test by yourself I think um the more you test you can find the different results okay so at this point we have mimic the official example and test it with LM St uh this is okay so I want to go to another example which I tested to actually I think uh we if we want to apply uh this uh uh package of uh autogen to other use case we need to have it in mind for example for me I would really like uh different agents to build a financial analysis thing to actually uh tell me if the stock is underpriced or overpriced and if I will I should buy it or not um I think there are many examples on YouTube you can also check but here okay I just uh uh test what’s uh what’s my initial uh uh construction of this team and to let them talk uh as a group so this code uh at the first part uh I also Define the different model but now I’m uh more specific I want to uh Define the mistal model that I’m using uh so just to uh make sure that I understand the the whole picture uh the other two I I leave it here but it does not make any impact if you want you can delete it okay so about the agents first of all uh I created more agents I create a team head ahead of the team uh the team head will do a lot of job I just describe them all here I think my code is correct but if I’m wrong I feel free to mention it um and here I actually uh describe the job it will do it will be the leader of the team of Stocker analyst you instruct the general analytical framework and ask analysts to process data and give investment strategy and suggestions in the end you combine opinions from both data analyst and the News analyst Okay I uh in this part I think the better you define the agent role the better it will run and get results in the later stage so I also have the analyst on data analyst on news so basically one will check the the data uh of the stock okay financial data I did not specify but if you it is recommended that you specify in a more detailed way I also ask it to actually analyze Trend plot a l graph and so on uh then about the analyst on news I ask you to check the news and also give some summary and also also provided the recommendation so in the end I want the team head to summarize so uh go to the next part uh the chat will initiate from the team head to the anal on Dat team head to anyl news Okay so uh let’s check the results maximum turn is two so I expect that will be two back and forth so uh yeah the message is analyze uh Apple stock okay so basically the te had actually first talk to the Anon data Anan data give some feedback I think this was uh the results we get from the uh model itself because it was trained a massive data set uh so actually the model actually figured out this by itself it it it U I think it have to be not 100% accurate so but you can see here it structured this um uh information I think very well you can see historical stock price earning per share and eventually it gave the conclusion that uh a solid long-term investment opportunity okay it looks very good and you can also try your local invironment to check the whole uh response okay also the uh if you check the LM Studio because you are running a server you can see the conversation and aut the log it is uh uh very uh satisfactory because previously I tried I if I remember correctly is the UI version of the autogen actually the configuration of the different agents and so of course you can do it in the browser in the web browser but it’s um I always meet some problem for example if the prompt is just one word uh and the feedback is just just two words it’s not working but in vs code with python J notebook it works well okay so let’s see first of all here uh the first uh response uh from analyst to to team head or okay then te head says uh actually the te head is working like the further more uh he actually uh tells the the analyst on data to pay more attention on the market developments and so on so is a it’s not like okay I get your advice and I summarize your advice and that’s it okay they actually want to dig more to explore more with uh analyst so yes uh analyst uh uh also I’ll give additional comments about uh add additional uh information about the stop price investment potential so actually it start it stops here okay so when I reach this point I realized that actually there is not another feedback so um the conversation uh is kind is is complete but uh if I want to get uh the conclusion from team head it does not give so I need to configure either the conversation terms or I change the description of the team head and make sure it works better okay so the is is still a prompt engineering as a basis so the better you engineer The Prompt the better you get the result so uh the system stop here and goes to the next conversation is a the team had to the analyst on news okay so at this point I realize it’s a separate chat okay so it asks the analyst on news to check okay and it says I’m a language model don’t have real time access of course so it actually uh it gave some general analysis based on the information it has for example like this uh information is in dated in 2022 so uh from the Pew uh conversational point of view I think it’s working yeah then I think head tells to the anal to news in addition to the analysis above it is crucial to consider both quantitive and uh qualitative factors uh in my opinion this is not very necessary because I would expect the te head to combine these two parts because uh the an onus is going to focus on the qualitative part and the analyst on data is going to focus on the quantitative part so that was my original design uh but okay so it ask this I think in the future I need to better refine my prompt and the description of the role uh okay so an news says okay it actually does the work it ask to not only analyzing the news but also to give some comments about the data and also examine the kpi and so on uh and also it did uh the competitive landscape man management efficiency and the industrial Trend comments okay so actually the analyst on the news is doing the job for the team head you can see here the team head should do the final comment by combining both the analysis you can develop a well strategy okay uh okay okay but eventually you see uh this one the model becomes very prudent because it suggests that you should consult another financial advisor so technically speaking it is not doing its job okay but you can see this uh flow of composition is going so you realize there should be some more uh adjustments either with the prompt with the max terms job description here uh s message I call it as job description okay okay uh so then I further refined the code I tried the um user proxy agent because uh I want to I consider user proxy agent is that uh the user inputs will actually influence the conversation so I set up this user proxy parameter according to the documentation of the autogen so you can see here uh there are some informations here like can name and so one so uh then now I start group chart so the group chart is going to actually involve multiple agents here for example I have three agents team head analyst on data analy on news and also I have uh Max around uh six because I think there are too many agents so maybe I need a uh longer conversation to get a meaningful results in the meantime I’m trying to to uh input my prompt okay so uh it’s not like previous version actually I write prompt in the code now I want the user input so I inut The Prompt okay the conversation starts you see the user proxy is the the user me will start the conversation uh and conversation to the manager and so the manager will go to the group chat in this sequence okay let’s see okay the system take my uh input as the uh user proxy to chat manager uh conversation and then analy on data okay says okay I’m doing the blah blah blah analysis okay and uh yeah it talks to the chat manager in summary okay uh Amazon stock blah blah blah okay it does not give a very uh precise conclusion because it says you needed to do further analysis so it’s not working very well and then uh it it says that it is important keep in mind the past performance is not question I think this is like a disc cler from the banks investment Banks yeah it’s it’s quite interesting uh yeah so then the group chat select the speaker failed to resolve the next speaker’s name I think there be some there are some issues with the naming of of the agents so if you see this kind of message I suggest you to actually check uh actually the next part the conversation goes to analysis of the uh of the news analyst of the news it says yes you’re probably right it’s important to remember historical data okay so it actually takes the as message of the disclaimer so it does not do much job okay and then uh it says absolutely providing clear okay it’s like uh uh not so useful right is a writing disclaimer so at this point you can see uh the group chat actually goes but I suppose there should be more definitions like either in my prompt or in the structure of the uh of the team and of the job description right to configure better these agents uh but as a start I think I would say this is um okay okay this model works and I need to further refine to actually uh do my uh do the jobs that that I intended uh to have them done for me uh yeah so this example stops here okay so just a quick wrap up you can do uh two agents back and forth conversation or you can have a group conversation okay so the next example uh for the uh models uh the definition are the same I did not change uh actually you can see like the previous example I ask the analy on data to plot to analyze a train of course it cannot okay so I needed to further work on it to see if I use some resource code to let the agent act on to uh draw the graph based on the data or I should uh I just should uh test the different Alternatives but at this moment I keep on going so uh the job description actually I uh I did some improvements some adjustments and I’m inviting a coder and a tester so uh in my opinion if you are going to invite a code and tester to a conversation and you want them to write the code and test code of course there are many use case that you can check on YouTube but as this moment if you only Define it here uh it will not work I’m going to show you uh so this part is the same we’re going to have a user proxy yeah so just just a reminder uh whenever you see there are some comments here you can stop the video and check some of the comments I leave the error message just for future check for example uh the original uh proxy code actually it asks for the darker environment I have never used it so I want to uh skip it okay so the I need to set here user doer Force if you want to refine this code I think first of all you need to understand the document of autogen of a specific function and if you uh have some difficulty to modify the code you can ask a chbt to help you but remember chat gbt has limited capability so uh the next part is still the group chat but here now we have the coder and tester okay next round six I uh manually insert my prompt okay and uh and here I think you need to pay attention because the system will recognize the whole part message should have two components okay a complete one not only like this the text information you can try because you are going to see the error message so I’m going to start a chart again okay analyze Tesla stock okay uh it’s the log actually says group chat select speaker fi to resolve the next speaker name uh yeah so there is the the some error but here the team Head Starts uh to ask the uh analyst on data can you provide me Tesla stock price I share recent news about Tesla uh and ask the analyst on news to share the news on Tesla both of them are working okay so uh first of all it promised that it will uh actually work and get the information yeah also the news uh Andy on news and think has a bit on information I would like to remember buying TLA stock at current due to the strong financial from earnings reports uh I would say uh I don’t see much data here for the team had to analyze for here the news yeah the news is positive okay a supp pass the expectation earning okay so H it makes sense but the capability of individual uh agents like I said before we need to enhance their capability uh so this is a part of the the conversation actually the CH manager actually uh talks to Team hat Team hat talks to to analyst then we go to the next part is the team head to the channel manager okay give a actually a response right it’s very structured I like it introduction this is a which company fundamental analysis valuation multiples I think if you further modify the code you can get uh uh quasi the the Quasi uh Prof professional report from this model okay then you can see here they also did the technical analysis and also news analysis uh I mean the accuracy because it’s not linked to the third party API not getting the news I think is uh uh not 100% reliable uh but the next attempt in the future I would try to like use a wi finance and or use the Google News AP uh Google use package or API to get such information to the model and let it analyze okay so in this moment we are always utilizing the knowledge base of the local logic language model okay so pay really attention to this if you try to have fun and to test the model yeah it’s okay but to really use it be very very careful uh then here uh okay you can see uh this is the different of from the previous one uh previous examples because we have more agents and uh actually we have modified some of the this job description of the uh of each uh agent so you can the results has some improvements I would say and then we go to the next part actually the team head now it’s your turn to play a role which of the following roles would you like to assume uh and what instruction question would you have for team members uh yeah so it well select them and provide the instruction to ask question accordingly okay so yeah now uh analy of data says dation is a important thing and so on this is quite General uh discussion now and the coder s me yes diversification is okay but the coder is not doing its job so there must be some problem with my setting okay uh in summary uh it suggest a diversification I mean uh the primary job of the coder is to code okay so actually uh okay let’s see here the coder start speaking again well set okay I’m not sure which agent actually talks to this agent before because I don’t see it here anyway you can do different test to check the workflow there is also very clear documentation of the autogen I recommend you to read it okay because I’m also still learning uh so in the next part uh I actually make my prompt very very complicated because I see the result is uh no so well I say analyze Tesla stock starting from the team head to get a general plan of the task then an analy on data to discuss with coder and tester to run a functionable coder to get analyze stock stock data let me see if I have a trunked uh version yeah is this one is easier to read analyze uh Tesla stock starting from the team head to get a general plan of the task analyst on data discusses with coder and the tester to run a functionable code to get analyze stock data and financial data and to give advice to the team head then blah blah blah so and now I give a very very clear instruction to the group let’s see how it goes so it takes a prompt okay and pass it to the chat manager okay and the team Head Start to talk back to the chat manager that I’m a team head so yeah how these uh agents will work okay it’s very well structured right so it first of all uh says that it will assign the task based on expertise of the team member three team member very well organized okay and since using two like broomberg wi Finance Fred uh I mean uh here I do not uh provide a Fred API or the Bloomberg API so uh this is a a very good uh uh assumption but not very realistic for me to implement a free version on the local environment uh then uh once they completed their analysis I’ll combine insight and I give a report yeah it’s very good so report the outline of the report it also describes is very uh comprehensive I like it uh but you see here group chat select speaker failed to resolve the next speaker’s name this is because the speaker selection call returned nothing okay so I think this problem uh repeats in the future I need to check uh is that did I misspell the name of the agents or there is other problem so let’s keep on going uh the next part actually I I think there is uh uh another solution to this kind of missing next speaker it will pick another speaker based on the chat there must be some logic behind if you want you can further uh check so basically now analysis on data I think is based on the the plan start to take uh to to talk with the major uh okay as a team leader I fully understand to I follow the objective so there is a kind of mix up because an on data is not a team leader okay he understands the job and starts to work as a team leader so but some of the part like this part uh yeah you you are actually repeating the previous conversation not very useful Okay so now the team head to the major the team leader my PR go blah blah blah Define blah blah blah I’m going okay sorry yeah here uh it describes his go and so on uh but not very concise not very useful to actually deep Digger uh deep uh deeper with the conversation uh then failed to resolve next speaker again so still this analy sound data chats with uh the chat manager okay so it did what was already there and he is the taking responsibility of the team leader okay uh but the from from the information of the step you can see there are also some some errors because here you see the from .1 to5 then from one to again so I think there is some problem with interaction with local RM because the description of the task is too complicated so the system cannot handle it I I see this kind of error sometimes before just during the conversation with MST model it will start to hallucinate is a different uh code no does not make any sense okay so you see here is it tries to make up a lot of numbers but they do not have any meaning so at that point the local Ser starts uh continues to run I manually stopped it okay so you see the error message here this is part of the server log you can see I take a screenshot the server log actually shows infinite numbers Okay so that basically says this project has to be stopped we need to change either the model either the structure of our agents uh so uh later I try another way I do not manually inserted The Prompt in this window I actually uh Define the message to to feed to the system uh yeah I think this is another way to feed the message to the system uh but uh somehow it actually only get the okay it ask my input if I remember correctly but you can try H this is not a very big issue you can either write a prompt here or you just uh manually input here okay so at this uh stage it starts a group chat again okay select speaker failed okay so at first it gave some comments okay so but here we have some improvements I will now step aside and let team head handled the analysis with using historical data it’s not 100% correct because I do not want the team head to do anything with the historical data so you can see there more complicated configuration with a more complex conversation uh that’s that will be needed so to the next part let’s see the team head to CH manager okay to provide the analysis I will do the combination for technical analysis okay it’s good post dat new analyst okay it’s uh it’s going well and it starts to okay fundamental analysis okay it tells you uh part of the information and also some subjective uh conclusion here but the problem is that uh the two analist have not said anything yeah so uh this is uh I don’t know how to define a hallucination or what uh here you can see the team head to the chat manager okay this claimer blah blah blah this is kind of a standard term again sentation okay subject to two what okay group chat select speaker failed again okay so if it failed actually the talking I think is like the LM one side output it does not say who is speaking so uh I will leave this kind of issue in the future to resolve and at this moment yeah this is uh actually the one of the result of the server log uh it shows a very messy messy picture so just to wrap up we have some issue remain for example this error code and the insufficient insufficient assignments of the job to different uh different agents and in the end I think we can improve in the future like I said uh The Prompt uh the agent job description I see it like this and also uh the detail configuration of the of the uh usage of autogen I think we have find also tons of example on the website the problem is that most of the example on autogen documentation is based on open API or other uh structure which you have to link to to the API of the online L model but if you are going to do with local LM Studio you always need to do some test to change the code to adapt to the local uh lar language model yeah I wish you find this uh uh tutorial useful and happy coding if you have any question feel free to leave a comment and I’ll I’ll reply and also to have a discussion with you so let’s advance in the AI agents creation have a nice day bye-bye
