# CrewAI + Exa: Generate a Newsletter with Research Agents (Part 1)

Channel: Alejandro AO
Video: https://www.youtube.com/watch?v=gXET04dJ66A
Duration: 1 hr 22 min
Language: English
Words: 14388
Transcript page: https://viewrankai.com/tools/youtube-transcript/gXET04dJ66A

---

[0:00] okay so we have the first one we have the first thought the first thought comes from the research agent we can see that he used the search and contents tool and what he thought in order to get here was to find the latest and most relevant news about the USA stock market I should start with a broad search query using the search and queries tool and the query that he searched for was search news good morning everyone how's it going today welcome back to the channel in today's video we're going to be talking about how to create a crew of AI agents that is going to create a

[0:30] newsletter for you this is interesting because we're going to have a team of AI agents that are going to coordinate themselves talk to each other and they're going to research the internet and give us a summary of the most recent news about any topic we want so that we can create a newsletter about that and send that to our subscribers okay it's going to be kind of an automation that's going to replace a very rudimentary team in our marketing department um so in order to do this we're going to be using crei uh if you have been following the channel for a while you already know how

[1:02] crei Works a little bit but in case you don't crei is basically just a framework that allows you to coordinate how different AI agents work together within a workflow so you're going to be able to have several AI agents each one of them with their own expertise and they're going to be able to work together in order to solve any problem we want in this case it is going to be creating a newsletter and we're also going to be using EXA which is a service that allows us to search the internet um in a very different way from Google and a way that is actually more useful for language

[1:41] models because they do not work on keywords but they allow you to search by meaning which is precisely how language models think so this is going to be super useful I'm going to show you exactly how you can build research agents using EXA how you can create tools for your agents that use EXA and yeah so I hope that you find this useful and oh yeah and by the end you will also have a graphical user interface that you're going to be able to use um so that you can use this yourself with your clients with your uh Team your co-workers Etc so yeah uh this

[2:19] is going to be awesome thank you for being here and let's get right into this [Music]

[2:36] [Music] all right so the first thing that we're going to want to do is to create our cond environment and in case you don't know what a cond environment is a cond environment is basically just a virtual environment that allows you to contain all of your python dependencies for a single project it will also contain the python version that you're going to be using for this project in particular in case you don't know how it is installed you can always click the link in the description get it installed and then continue with this tutorial uh so the next thing that we're going to do is to

[3:09] cond create and then we're going to do D- name and then we're going to name our content environment in my case I am just going to name it crei and then you're going to say python equals and whichever python version you want to work with going to hit enter and I have actually already created this content environment but to you it might take a little bit longer uh just follow the instructions for that uh once you have it created you're going to be able to do cond activate and then call the Conta environment that you had previously created by the name that you gave it like right here so in my case I named it

[3:46] crei so I'm going to just activate crei and you should see somewhere within your terminal that your new environment has been activated once that it is activated what we're going to do right here is to install crew AI so I'm going to do pip install crew Ai and it's probably going to take a little bit longer for you because I already had it installed but once it is installed uh just by doing pep install crei you're going to be able to use the crew crew AI command that is going to allow you to create a project uh with all the project structure that you're going to need and in order to do this

[4:25] you're going to do crei create and just name your project in my case I'm going to name name it newsletter gen like this and then as you can see it has created this new folder called newslet gen and it has created all the folder structure that I'm going to need to bootstrap this automation okay so let's get right on with that opening it with code newsletter gen all right so here's the directory folder that was created for us as you can see we have a source directory with the name of our Automation and within it we have a config directory and a tools directory the config directory contains

[5:11] our agents. yaml file which is basically just an example of the agents that we could build here's an example of a researcher and as you can see we have the role the goal and the backstory for each of this and this is very important to improve the results of your Automation and here we have the test t as well we have two examples of tasks that we can build here is a research task and here we have the description and the expected output I am going to give you in a moment more uh more detailed explanations of of how to write these descriptions but just keep in mind

[5:47] that this is the prompt that is going to be sent to your language model in order to complete each task okay then we have here an example of a tool that our agents could use to to complete the tasks the crew. py file which is the main class that we're going to be working with it contains methods which can be your agents or your tasks and then the final crew that is going to be generated and then right here we have the main.py file which is our uh which actually just runs the automation okay then just all these config files. dnv for your API keys at an empty test

[6:26] directory and there you go so now what I want you to do is to actually understand how to plan these agents and tasks before trying to populate them right here because you may find that if you jump into populating these tasks from the beginning without proper planning your automation is most likely not going to work so you have to have a proper planning on what you want to automate and how you want to automate it before jumping in and completing this part right here and that is what I'm going to show you right now how to do how to plan uh an automation with creai all right so what we want to do right

[7:02] now is to see how we can plan our Automation and just to be clear for this example we're going to be creating an automation that will generate a Weekly Newsletter with the most recent news about a given topic okay and it will output the HTML code with all of the contents of the newsletter ready to be sent by email okay so um in order to do this we're we're going to be using crew AI which is the framework that will we will use to orchestrate our team of AI autonomous agents as you can see right here this team is going to work for us in solve and complete different tasks

[7:44] that will take us to the final result that we expect from this automation okay now in crei as you might remember there are four main abstractions well three main abstractions in this case which is tasks which are the steps to complete to complete complet your automation agents who are the AI agents or the entities who will actually complete the tasks these are um you can think of them as your employees and the tools which are the functions or the services that your agents are going to need to use in order to complete the tasks okay now when I do my planning for these kinds of automations I like to divide my planning

[8:23] into four different steps first I Define the input and the output that I expect from my Automation and this is very important because of course you want to be consistent in what you will give to your automation as an input in this case it is going to be the topic of my newsletter it can be uh recent news about AI recent news about the USA stock market and the HTML template that I'm going to be using for my newslet for my newsletter this is of course important because I want that every time that I run this automation I will get the exactly same uh design okay I want my

[8:57] design to be consistent in other words I do not not want that the first time I run this automation the font is going to be times NE romen and the second time that I run this automation the font is going to be something like um comic sense okay so that is the first step to define the inputs and the outputs then the second step is to define the tasks that will take you from your inputs to your outputs and you can think of it as the tasks that you yourself would do if you were the one uh performing this Mission okay now in our case since we're

[9:29] going to be creating or generating this newsletter remember that if you were doing this probably the first thing that you would do is research the recent news about the given topic then summarize those news as you can see that is also part of this of this task and then with the summarized version of this news you were probably going to verify that they work correctly you're going to order them in order of relevance and then lastly you're going to complete your HTML template with the new stories that you found okay okay so that is the tasks that you would follow and that is the second part of planning this automation

[10:06] so first one was defining your inputs and outputs and second one is defining the tasks that will take you from your inputs to your outputs okay now the third thing that I usually like to do is to Define my agents once I have my tasks okay so once I have my tasks pretty well defined the idea is to create the agents and think of your agents as the members of your team who will be in charge of this automation and try to describe them as you would describe a team of humans think of what you would like your employees to have um what skills you would like your employees to have if you

[10:40] were uh delegating this task or this automation to your team and in this case as you can see we have a researcher who will conduct a research for our news a chief editor who will verify the news and reorder them and also provide some insight about them and our HTML composer or developer who will populate the HTML template with whatever the editor and the researcher give him okay so that was the first step and the last step that I usually like to do is to define the tools that my agents are going to need to use in order to complete the tasks okay and remember that our agents are powered by Ai and by

[11:19] that I mean that they are powered by language models okay which means that they can recursively call a language model until they have a response that is decent or something closer to what you would expect and they can also ask questions to each other in case they feel like they need help from one of their colleagues that is also why it is possibly a good idea to have agents lying around not assigned to any task because when you have a team of Agents a given agent who is assigned to a task can ask a question to anyone even if they are not uh specifically assigned to

[11:54] a task so you can have another agent right here who can be I don't know a copywriter or something like that and the researcher might ask a question to them if they need some copywriting for example and as you can see right here some of our agents are going to require access to external Services right because if we do not give them access to these external Services all they can do is recursively call the language model or ask each other which will also recursively call the language model so we have to create tools for them to actually have access to more sophisticated um services in this case

[12:28] the tool that we want to create is a tool that will allow them to search the internet for recent news and in order to do that we're going to be creating a research tool using the EXA service provider and I'm going to tell you a little bit more about how EXA works but just to summarize it very very quickly it is a search engine that works by meaning rather than by keyword which is very useful because language models usually think by meaning rather than by keyword okay and we're going to be creating three different tools the first one is going to be search and contents that

[13:02] will allow our agents to search for a query and get the results from the search results and also the contents from that page then find similar we'll take uh URL and we'll return similar websites that contain similar content to the contents of the website of that URL and then lastly we will have get contents which will get the contents of a given URL okay and these tools we're going to assign them to the research and I am actually also going to assign them to the chief editor so that they can also get contents and find similar websites just in case they need them okay and oh and I forgot to mention I

[13:41] remember that I said that the agents are going to be able to talk to each other most of them are going to be able to do that except for the HTML composer or the developer right here because remember that when we were creating it or when we're going to be creating it we're we're going to set the allow delegation to FS because we don't want them to be talking to their colleagues we only want this guy to be focused on just replacing the contents that the researcher and the chief editor are going to give them within the HTML template okay so those are the three abstractions that you're

[14:15] going to need to plan for your automation something useful that you can do as I mentioned in the previous video is to brainstorm your idea to CH PT and to ask it to give you the tasks and the agents and the tools that you might need that will however will not get you super far but it will give you a head start especially if you're using the GPT from crei I'll add a link in the description to that GPT but to use it of course you need to have uh GPT plus so there we go now that we have defined all of our elements of our automation we can

[14:43] actually start creating the prompts within our our configuration files okay so let's do that right now all right so here we are back at vs code or whichever ID you prefer to use and the first thing that we're going to want to do now is to populate the tasks okay we have already planned our automations now it is time to actually write the prompts that we're going to be using okay in the previous video I actually showed you how to bootstrap this prompt writing by using the GPT from crew AI so you just brainstorm your idea to the GPT and it gives you some prompts some descriptions and expected

[15:28] out that you can use however those are usually not completely optimized it was going to be up to you to optimize them via trial and error okay and that is actually what I have already done right here I have already uh started off with a few prompts I have optimized them via trial and error until I found that they work correctly and what I'm going to do is I'm going to replace this right here with the prompts that I have already created and I'm going to explain to you some of the characteristics of these prompts so that you can actually build your own and improve them to make them

[16:06] as as good as possible okay all right so as you can see right here I have the three different tasks that I had already planned the research task the edit task and the newsletter task and let's take a look at the research task as you can see the description is pretty long the expected output is pretty long so let's take a look at that so here you have the description here it says conduct a thorough research about the latest news on and then the variable that I'm going to be using right here is going to be between curly braces because this variable is going to be sent as an input

[16:40] to my Automation and it is going to be populated in the prompt whenever I have thisly braces like this okay be sure to look for sources that are reliable and publish recent news do not include articles that are not news material or that are not directly related to my topic with this research compile a list of the most relevant news stories that you found okay so this is a very very specific and very clear description of what I want my agent to do now a good technique of something that you can do after that is to give it some rules to follow okay and something that you will

[17:18] find yourself doing is updating your rules on every iteration of trial and error in order to correct for common mistakes that your agents are doing so in this case we have only include articles that are specially related to topic and in this case as you might imagine I added this rule because my agents were returning top um were returning articles that were not necessarily relevant to the topic in question then we have do not include sources that are not a news article if the content of the page includes a list of Articles or looks like the front page of a website do not include it in the

[17:56] list now as you might imagine the this one was also to correct for a common mistake from my agents in which they were actually returning lisal articles I mean lisal Pages which had several lists of Articles um in them some of them which were not relevant to the topic in question so that's why I added this one right here so that they can get rid of those summarize the news in a few sentences make the summary as long as POS as long as necessary to include all the relevant information but not too long for a news later now this one I added it afterwards because as you might

[18:31] remember what I told you before this one was actually a separate task and so I realized when I was uh running this in one of my trial and error iterations that the list that was returned from the first research task was already pretty good and already had a pretty good summary of what the news was so I just asked you to complete the summary right here and that was all that I needed to do to remove the Redundant task of summarization next we have include the URL of the article where you found the news include a minimum of seven news articles and a maximum of 10 news

[19:11] articles in the list and I will show you later how this affects the automation but as you can see this one was also to correct for some problems that the agents were having when using the research API okay so there you go this this is a good Tech technique that you can use when you're creating your automations and you're creating the prompts for your agents to follow now here as you can see I have some extra instructions about using tools and this one was as well to correct for some common mistakes that some of the models were having all right so that was the description very detailed and with a set

[19:51] of rules to follow now I want you to take a look at the expected output right here here as you can see we have just a description of the output U marked on document with the most relevant news stories each news story should contain the following then the title of the news the summary of the news and the URL of the article where in news was found now here is something very interesting and that I would recommend that you do every single time that you're running an automation like this this is called one shot or few shot example and it is a prompt engineering technique that

[20:20] consists of enriching The Prompt with some examples of what you want to get uh from the language model completion okay so right here we say here's an example of the format of a use article that you could include in the document and here you have the title and a real title and the summary and a real summary of what it would look like the URL and a real URL so you can of course add several examples like here I only added one but yeah this is a super good technique to make sure that your results are consistent with what you expect them to be because language models are known

[21:00] for being very um non-deterministic so this can help you improve that um deterministic element um significantly okay so I would encourage you to add these examples and just a note on the mark down markup TXS right here there they are of course not necessary but I have found that they do help the language model identify when a certain part of the prompt start starts and when I it ends okay so here's just example tag and end that example tag all right so now let's actually take a look at our agents and write the prompts for them I am not going to be writing the entire prompts in front of

[21:45] you because it is a very lengthy process and just as with the tasks it also requires a lot of trial and error to get them right but I am going to paste the ones that I have right here and I'm going to explain them to you okay so here as you can as you might remember from the planning stage of our automation we have three different agents we have a researcher an editor and a designer or I mean you can name it whatever you want which is the one that's going to be populating the HTML template for our newsletter the researcher will search the internet the

[22:20] editor will um make sure that whatever the researcher returned is correct and also provide an insight on that and then the designer is going to populate the HTML template with whatever the editor returns okay now let's take a look at the elements from the researcher one here you can see that we have a role a goal and a backstory in this case the role is a senior researcher this is going to be the role of our agent then we have the goal of our agent in this particular project which is going to be uncover Cutting Edge developments in and here we have the topic which is the

[23:00] variable that is going to be interpolated from our input variables okay remember that those input variables are supposed to be in curly braces like this and then right here we have the backstory now this is a very interesting part of it and the idea behind this remember that you have to think about this like a manager you have to think of the backstory of the employee that you would like to have uh performing this task okay so here for example we have a researcher and it says you're assistant journalist with notes for news you're known for great research skills and the ability to dig up the most interesting

[23:37] stories your reports are always thorough and well research researched making you a trusted source of information you always all follow the rules and guidelines provided to you this is of course uh making reference to the rules that we wrote in the tasks prompts right here the rules and and you never forget to include the complete URL of the article where you found the news and this as you might be guessing the this is to correct for a common error that I was having where the actual URL was not pointing to the right um article in the web okay so there you go remember that in order to

[24:20] write a correct backstory you have to think of it like a manager and you have to think of the backstory that you would like your employee to have if it was your employee performing this task okay we have the same for the editor and for the designer so there you go that is how you create the prompts for your agents now let's actually Implement them in our class for our crew okay so now that we have our agents

[24:49] and our tasks prompts we can go to crew. py right here and we can start populating this placeholder file that was generated to us when we initiated our project okay here as you can see you have the main class which is going to be named with the name that you gave to your crew in my case it is newsletter gen crew and right here you have The Decorator which is going to help you to automate a lot of the things that you would otherwise have to do by hand okay here you have the first property which points to the configuration file for the agents this is of of course the file

[25:28] that we just finished uh editing which is this one right here in which we defined our agents prompts now the next one is the tasks config which also points to tasks. yaml right here which is the configuration file that we just finished editing as well then you have a couple of placeholder methods decorated with The Decorator agent which is imported from crei and right here these are the methods that you're going to replace with the agents that are in your agents. yaml file same thing for your tasks then of course you have some parameters right here and last but not least you have the crew method which

[26:09] will return an instance of the crew object which is the one that is actually going to um run the automation with all of your team of Agents okay so here you have of course that it takes the agents as parameter the tasks as parameter and these are of course automatically this I mean these properties are automatically uh created by the decorators and here you have the process in this case we're going to be using sequential which just goes from task to task uh you could also add hierarchical right here but that will require you to add a manager llm which would basically create kind of a manager for your team

[26:55] and this manager would assign the the the different tasks to different agents depending on the expertise of each agent this is a little bit more complicated so we're not going to be focusing on that and I feel like for most use cases sequential is the most useful um process that you want to use um just remember that if you are going to use hierarchical and you want to you want to test how the manager llm works here you can pass in any instance of a language model instantiated using the blank chain classes all right so you can use chat groc open AI chat Etc okay all right so

[27:34] let's start with the first one right here the first one is going to be our researcher so we're not going to change the name for this one and as you can see right here it says that the configuration is going to be getting it from Agents config and the key inside agents config is going to be researcher and let's see what that means researcher is this key right here so that means it's going to be accessing the information within this key in our yaml file now in case you haven't watched the previous video where I explained this um what is happening here behind the scenes is that this decorator is taking the FI

[28:11] the yaml file right here and recreating the property agents config as a dictionary okay so here it might look as though you are indexing uh string because this one right here is a string but you're actually indexing a dictionary so that's okay but that's happening behind the scenes all right so this one right this line right here will assign the role the goal and the backstory to our agent we have to set verbos true in this case in order to actually see in our terminal what our agent is thinking behind the scenes and this one right here is just a I mean the tools parameter is just a list of all

[28:51] the tools that your agent is going to need to perform the actions we're going to add them once we have coded those tools okay so that's for the first agent right here let me copy and paste the second and the third agent which correspond to the editor and the designer right here all right so I'm going to delete this one right here and I'm going to replace them with this agents that I have created right here so as you can see right here oh I am missing a decorator right here so as you can see I have the editor and the designer which are pointing to the

[29:25] editor key right here and the designer key right here okay uh here I have the tools but I haven't created them so I'm going to leave that empty for now and there you go so you have the configuration that's been added like this and for the designer or the HTML uh composer I added another parameter right here which is allow delegation equal false and the reason for this is that in crew AI every agent has the possibility of delegating their work to any other agent or asking questions to any other agent whenever whenever they are stuck okay now that is usually a very good feature however some agents have some

[30:14] tasks assigned for which they really cannot get any help from anyone else and that is the case of the designer or the HTML composer agent which is this one right here uh his job is only to replace the contents of a HTML template there is nothing that the editor or the researcher can do to help him so by setting this to false I force him to finish the task by himself without having to talk to to their colleagues and it is important for you to identify the agents that are have very specific tasks for which they cannot get help from anyone else because know that when

[30:55] your agents have the possib AB ility to talk to their colleagues they will most likely do it and that of course is going to cost you resources because it is additional calls to the API so make sure that you set allow delegation to fals if you are sure that the task can be completed by the agent itself so there you go now you have the three agents right here and now it's time to add our tasks now the first one actually looks very much much like this one we have research task and it's accessing the research task key from our yaml file now let's take a look at how that one looks like

[31:39] research task here we have the description and the expected output for this one so this is correct and each task remember that I told you that each task has to have at least one agent that is assigned to it because it has to be performed by someone and in this case we are assigning the researcher agent which is up here to complete it all right now for the second task we're actually going to be replacing this so let me copy and paste that from here all right so here this one right here I'm going to be replacing it for this ones right here got a lot of

[32:18] python and its indentations okay so here you have task The Decorator remember that for every task that you create you should decorate it with task if you're using this project setup if you're using just the crei abstractions on your own of course it is not necessary to add these decorators but if you're using this setup it is a good idea to use them here you have the second task which is edit task and as you can see it is accessing the edit task key from my task. yaml and it is it is being assigned to the editor and last but not least we have the newsletter task which is the one that is

[32:55] going to be populating the HTML template and it has the configuration from the newsletter task right here okay and as you can see we have created our three tasks each one assigned to a different agent uh remember that you could assign two or more tasks to a single agent but in this case we're using different agents for each one of them all right so congratulations we have successfully created our agents and our tasks now the next thing to do is to create a tools that our agents are going to be able to use in order to complete the tasks because remember that so far all that our agents can do is to query

[33:37] the language model recursively that is to say that they can think recursively until they find the the required answer or they can also ask questions to other members of the team that is to say to other agents the only agent that cannot ask questions to its colleagues is the HTML composer or for developer because remember that we set that setting to fals while we were creating it so in other words all they can do is ask questions to each other and generate text if we want to give them more sophisticated capabilities like accessing external Services we have to create tools for them to use and that is

[34:17] what we're going to be creating right now we're going to be creating a research tool set that will allow them to search the internet and read the contents of those results I'm going to explain a a little bit more about the um individual Tools in a moment but first I want to tell you a little bit more about the service that we're going to be using to create these tools and the idea right here is that we're going to be using EXA so very broadly EXA is a semantic or neural search engine this means that it allows you to search for information on the web in a different and probably more

[34:52] efficient way you can think of it as an alternative to Google uh but instead of searching by keywords it will search by the meaning of your query okay now what does this mean let's take a look at that I have some explanation right here in the in the blog post now as mentioned before we are going to be creating a research agent and this agent should be able to find the most recent news on a given topic right for our newsletter this topic can be the USA stock market artificial intelligence or whatever you want so let's think about the query that our language model is going to generate

[35:29] in order to use our tool right because remember that our tools are essentially functions python functions that they can call and they take a parameter and this parameter in this case is going to be the search query okay so if our research agent is going to be using this tool it is going to generate a query that is going to look something like this latest news on AI or latest news on the USA stock market right in order to find the most recent news news on that topic now let's take a look at what happens when we look at that on Google let's see so latest news on AI and you can see that

[36:06] these are the results the first one is the sponsored one we can probably ignore it but you can see that the other ones are also front pages of other blogs or other newspapers and something important about this is that they contain the word latest news on AI which means that they are optimized to appear on the first results based on the keywords of your query okay and that is a problem because if our language model looks for this query it will only find the front pages of other websites now let's compare this with a semantic or neural search engine like EXA now EXA has a graphical user

[36:45] interface as well that allows you to use their API through their website I think that you can use it right here there you go and let's take a look at what happens when we look for latest newon AI so so let's do that latest news on AI and we can filter by time I'm going to say that we're going to work for the past month for example and we're going to click on search as you can see the results are the actual news because it is searching by meaning so we're talking about the meaning of the query okay and this is most likely going to be much more useful

[37:23] for our language model and this is of course because our language model models think on meaning rather than on keywords now that's how this works and their API works pretty well and they have a th000 requests per month for free so you can use them and so we can actually now start creating these three tools that we will be creating using their API okay so right now we're going to create the tools for our research agent and we're going to be using exit to create these tools okay so we have the placeholder right here that was generated by us um by the crei CLI for us um and we're just going to populate

[38:03] it with our own classes in importing EXA okay so once that EXA has been installed either with poetry at XA py or with Pip install XA py you can do from EXA we're going to import not EXA like this but EXA like that and here you have the placeholder for what a tool would look like um here we're going to name it search and contents because this tool is going to return the search results and the contents of the search results and here the first property is the name so we're going to name it search and contents tool and then the description remember that it has to be a very clear

[38:49] description of what your tool actually does because this is what the agent is going to read and this is what it's going to use in order to decide whether or not it has to use this tool to complete the task in question okay so in other words if the task is a research task it will be able to read this description that this tool searches the web based on search query for the latest results the results are only from the last week it uses the EXA API and it also Returns the contents of the search results so this is a very clear description of what the tool does and

[39:22] now your agent will know that it has to use this tool if it wants to search the internet okay and then right here we have the method the main method of run which is the one that is going to be executed when we when the agent wants to use this tool okay and any parameters you want to add to the tool you're going to have to pass them as arguments right here so in our case the parameter that our agent is going to pass is going to be the search query and right here let's just add the implementation of EXA so the first thing to do is to initialize

[39:54] our EXA client like this and this one actually takes the API parameter so let's import it from the environment variable called XA API key let going to come to umv and as you can see I have already added my EXA API key variable right here I'm just going to populate it in a second when I get it from their dashboard so I'm going to come right here I have to import OS for this to work and there we go so now let's actually get the search results from this client so we're going to going to we're going to say that the search results are going to be EXA do search

[40:33] and we're going to tap into the method search and contents which will return the search results and also the contents of those search results okay now you could pass only the query like that but let's add a few more parameters to show you um the flexibility of this method so the first thing as you can see right here we have to say what is the query that we're going to be using um and we're going to be sending the search query that our uh agent is going to formulate then we're going to say use Auto prompt to true so that because remember that XA works on a Transformers

[41:08] architecture so if for some reason your agent does not formulate a query that works good with EXA EXA is going to reformulate the query to give you the best results okay so that's what you get use autoprint to true and then let's actually say that we only want the search results from the previous week so in order to do that I am going to add a start published date and I'm going to create a date Cod of that equals to the previous seven days and in order to do this I'm going to import from daytime I'm going to import daytime and time Delta and let's initialize right here

[41:46] the one week ago date which is going to be equal to now minus at time Delta of 7 days and the actual date cut off which is going to be equal to that date in the format year month day which is the one that EXA likes for this okay so let's now here just say that this start is going to be the date cff like this and right here you can send a couple more parameters that I want to share with you and might prove useful at some point which is the text parameter right here here you can specify several uh parameters that want you can for example

[42:29] add include let me just move this up right here you can for example add include HTML TX and what this is going to do is this is going to return I mean if you set this to true this is going to return the HTML tax of the content so it's going to return the P TX the a tax and if the search results contain a table this is also going to return the HTML text of the table so this might prev for some kind of results probably not for all and maybe it will make your results too verbose but if you're looking for table like content this

[43:05] might Pro useful to set this to true just going to set it to false for now which is the default and something that is also pretty important is Max characters I don't remember what is the default value for this one but in my experience while I was doing some tests for this I found that the contents of the search results that this method was returning learning were too short so I wasn't actually getting enough information for my search results so here I just said that I want 8,000 characters for each search result um and that is going to give me enough information for my agents to actually

[43:42] summarize the new story okay and then with those search results we can just return them right here there you go so now our first tool is pretty much finished and let's create the other two tools which are find similar which is going to take a URL and return similar websites that talk about the same topic as that URL and get contents okay I could create separate files for this but I'm just going to create the three classes inside of here so I'm going to say find similar and just like before let's see what co-pilot tells me so find similar tool it's going to be the name and the

[44:24] description is going to be searches for similar articles to a given article using the EXA API takes in a URL of the article there you go and then right here we're going to define the run method which is going to take the URL the article URL okay so now just like before we're going to set this to one week ago we're going to set Delta under date cff and then right here we're going to also initialize a EXA client I mean we could do this as I did in the previous video using an EXA uh method but this works pretty good and let's say that the

[45:06] search results are going to be EXA Dot and we're going to tap into the find similar method which will return the similar URL keys so right here let's pass in that the first parameter is going to be the URL which is the article URL and this start publish date is going to be date artical cff because we do not want similar articles from two year two years ago we want just from the previous week so there we go and then we can just return our search results now let's create the third tool which is going to be get contents and in order to do this we're also going to

[45:47] initialize a class we're going to call it get contents like this and it is also going to be based on the base tool imported from C tools the name is going to be get contents tool and the description here we have U detail description let me copy this for you so the description have it right here it's going to look something like this gets the contents of a specific article using the XI API and it takes the ID of the article in a list like this okay so here I provide an example of how the language model or the agent should send the parameter and as you can

[46:26] see see it has to be an array or a list of strings and each string is going to be an ID of the articles for which we want to find the content okay so here you could have a second URL Etc now here please consider that we are saying that the agent should pass in the ID however in practice the ID is actually the URL in the XI API so just keep that in mind so once that the description is finished we can actually Define the run method so let's see the article IDs is going to be the argument that we're going to pass and that article ID is going to be the

[47:13] URLs and that article ID is going to be a list and it's going to out this method is going to Output a string so just like we did before let's initialize our XA client by doing like this and just calling an our environment variable for the API key and right here let's just say that the contents are going to be equal to XM we're going to be using the method called get contents like this right here okay and this get contents method as you can see it takes in either a string or a list of strings as we specified right here and there we go actually I'm just going to say that here

[47:50] it can be a string because a language model will always pass in a string at the beginning so there we go so now we can pass in the article IDs and we can return the contents like that all right so now to get the EXA API key and in order to actually test that our tools are working we're going to go to xa. aai and once you have created an account we're going to go to dashboard and right here on the overview tab you have your API keys if you don't have any you can create one just going to copy it and let's go back to visual studio code to

[48:26] to add it to our environment variables okay so we're now back at our project right here we can go to tnv and here we are going to paste our EXA API key like this I'm of course going to disable this API key before publishing the video but just to let you know that here's where you can put it now let's go back right here and let's actually start testing these tools to see if they actually work so in order to test them I am just going to execute them directly so I'm going to do if name equals Main and I'm going to create initialize the search and

[48:59] contents tool and I'm going to run the Run command and the search query that I'm actually going to pass in is going to be latest news on the USA stock market and then I'm going to print the results there we go so now let's open the terminal let's clear this right here and let's execute this

[49:28] so there you go here we have the results as you can see let me just open this a little bit more here as you can see we have that it used the search and contents tool here we have the title of the first result here we have its URL its ID which as you can see it's the same as the URL the score which is how relevant it is to our query and here you have the contents of that result okay uh same thing for the second one third and we have a bunch of results that might prove useful so the first tool is working correctly now let's

[50:01] actually copy one of the IDS right here and pass it on to the other tools so I'm going to close this and let's actually initialize the other tools so let's say that the other tools are going to be find similar and let's run print our similar are going to be find similar and we're going to run that the article URL and I'm going to past the ID of our find similar uh URL that was retrieved by the previous method just going to comment this one out and let's print this one right here and see what it returns so let's execute this and there we go we have some other

[50:50] similar articles and as you can see these ones don't actually include the content so this is an instance where the get contents tool might be useful for us okay so here are similar articles that contain similar contents to the news that was retrieved in this URL so the second tool is working correctly now let's test the get contents tool and this one remember that it takes in also the article IDs in a list so let's do that right now so what we're going to do is we're going to say first going to initialize and get content tool and we're going to call it like this contents are going to be equal

[51:30] to our get contents we're going to run and we're going to pass in our ID which is the URL and let's print the contents of this URL right here so let's see if that works I'm going to click on enter actually it is yeah it's using the fine similar as well because I forgot to comment that out but as you can see here it used the get contents tool for the URL that I pass in and it Returns the contents and just to be clear this is not actually scraping the website this is retrieving the contents that are already within the EXA index okay so XA

[52:09] is not necessarily helping you to scrape the data it just already has scraped the data itself okay so that's why it's pretty useful so you don't have to code your own web scraper in order to get the contents of the websites retrieved by EXA so there you go apparently our tools are working correctly we can go back to our crew. py file and add them to our agents right here and in order to do that we're going to have to first import them so we're going to have to say that from newsletter gen do tools. research we're going to be able to import search and contents find

[52:48] similar and get contents which are the names of the three tools that we have created and now what we can do is is just add them right here like that and remember that we said that the editor was also going to have access to this tool so I'm just going to add them right here like that so search and contents find similar and get contents there we go so now that our agents have the tools we can actually start testing our crew to see how it works so let's do that right now all right one quick thing that I wanted to add right here is that when

[53:27] adding the tools I made a mistake right here actually you have to initialize them when you are importing them so right here don't forget to add the parenthesis otherwise when you try to execute this this will not work okay now once the DAT is fixed we can continue with the video okay pretty good so now we have successfully created all of our tasks our agents and our tools now we're going to just add our inputs to our Automation and run it to see in that everything is working correctly okay so in order to do that actually remember that we had the topic of the newsletter as an input we

[54:04] had the HTML template as an input and I'm actually going to be adding a third input right here which is going to be a short personal message that we're going to be adding to the top of the newsletter you're going to see how that works in a moment but just remember that these are inputs that we don't we haven't yet uh defined so we are using them right here in our prompts remember that everything that you put in between these curly braces is going to be interpolated with the inputs that we're going to pass in but also note that right here we have another two input variables which is personal

[54:42] message which is the one that I told you that I was going to be adding real quick and the HTML template which is the one that is going to allow us to have a consistent design every time we run the automation so in order to do this I'm going to go right here to my config directory and I'm just going to add a newsletter

[55:03] template. HTML and right here I'm just going to add in some HTML code that I had already prepared here you can see that I have already defined the Styles and here you can see that I have the H1 for the title of the entire newsletter here I'm going to add the personal message that I told you about so that I have a personal a title a personal message then a list of all of the relevant stories each list is going to have a H2 for the title of the story the summary of the story why the story is important and a couple of links or the sources for the for the story and

[55:39] there you go so that is the template and in order to actually pass in all of this as an input what we can do is go right here to Main and execute our automation from here and what this file here is doing it is basically just importing newslet crew from crew. py right here it is calling the crew method which as you might remember right here returns an instance of crew and on that instance of crew it is running the kickoff method which takes inputs that are going to be interpolated remember that the Ki of method all that it does is it runs your automation okay so what we can do right

[56:18] now is just update this inputs dictionary with our actual inputs now since we're going to be testing this I am just going to say that this one right here is going to be an input for the user in the terminal I'm going to say that enter the topic that you want your newsletter to be about so enter the topic for your newsletter then the second one is going to be the personal message and this one as well let's let's say that it is going to be an input enter a personal message for your newsletter and very importantly we are going to pass in the HTML template now

[56:59] remember that the keys for these variables right here in the dictionary have to be the exact variable names that you are using in your prompts so here you have HTML template and here you have HTML template okay here you have personal message and here you have where is it here you have personal message okay so just be sure that you are using the exact same variable names because otherwise it is not going to work and here we're just going to load the contents of this file right here and in other to do that let's just create a quick function right here let's say load load HTML template and it is going to return

[57:46] with open and the path is going to be SRC newsletter gen config new newslet template R in order to be able to read it as file and the HTML template is going to be equal to file. read and then we're just going to return HTML template like that okay so this function is going to read the contents of this file and it's going to return them as a string right here and this is exactly what we want I am going to run here load HTML template like that great so now now that we have added all of our inputs right here let me just be sure that this is saved we

[58:29] can just go to readme.md and make sure how to run this using the framework that CI suggests that we use which is using the Poetry scripts which work pretty much just like npm scripts as you can see right here here is a poetry script already created called newsletter gen and what it does it basically goes to newsletter gen. main which is this file and executes the method run which is this one right here okay okay all right so now time to test this and before we do that we have to make sure that our API keys are correctly set up as you can see right here I have already added my

[59:05] open a API key don't worry I will disable this before uploading the video in case you do not have an API an openai API key you can go get it at open.com and in case you do not want to use a paid API like open AIS don't worry in a few seconds I'm going to show you how to do this using pretty much any model you want want and this other environment barel right here can be pretty useful for you especially if you're using the default configuration of crei because I'm pretty sure that crei by default points to an older version of gp4 so this one right here will'll set this one

[59:42] to the latest version and you can of course change this to any to the latest model at the time of watching okay so once that that is set up remember that in order to run all of this all you have to do is do poetry run and then the name of your project and in this case I'm going to do it like this let me just open this a little bit bigger we have a um deprecation warning hopefully it's not going to cause any big troubles and here you're going to say the topic of your newsletter it's going to say that it's going to be the USA stock market

[1:00:18] let's make a USA stock market a financial newsletter and a personal message hello here is the news for your week there we go all right so I'm actually going to pause the video right here and I'll be back once the run is complete because it might take a few minutes to finish okay so as you can see the automation is finished let's actually take a look a little bit at what it actually did so as you can see here um it checked some some contents of a website apparently it searched more the internet communicated then there you have the thought now I've verified the URL so actually here it was verifying

[1:01:06] the URLs now that I verified the URLs lead to a proper articles I need to enhance the the titles for better engagement provide context and reorder the stories here you see that it made a mistake and something that is pretty useful about these kind of automations with AI agencies that when they make a mistake they can correct it by themselves as you can see right here it made several mistakes while trying to use a few tools then it went on and said I need to enhance the titles and use and then it went on to give a final answer and here we have the final answer for

[1:01:37] our most recent stories so there you go these are the recent stories and then down here we have the last task which as if you might remember from The Prompt that we have right here remember that we have follow these rules Etc and here we had the HTML template that we were passing to it so that's exactly what's going on right here here um I think it is right here yeah fill the following HTML template with exactly the same information that is given to you Etc and then I provide the HTML template right here and then I can now give a great answer and it Returns the HTML okay I actually forgot

[1:02:19] to save the output as a file so what we're going to do right here is we're going to copy this right here and let's actually just create a new file right here just for test I'm going to call it test. HTML and I'm going to paste the HTML that was generated and see what it looks like on the previews so let's serve this right here and it was made right here okay so here we go um seems to be okay we have weekly stock market op dat a searching uptimum then we have our personal message right here and then we have one story for each relevant story that our agents found uh

[1:03:07] let's see if the links work I click this one seems to be okay perfect let's see nastic climb has tag semiconductor stocks rally there we go okay pretty good well apparently our newsletter was generated correctly now let's add a little bit more features to this automation first of all let's automatically save the generated news uh newsletter file into a file not only log it on the terminal and secondly let's also add the possibility of using several language models not only gp4 okay and last but not least we will be building a graphical user interface for for this application so let's do that right now all right so now we have

[1:04:00] finally finished this automation that looks pretty good uh we have successfully created our agents we have successfully created our tasks and we have our crew that is working correctly we already run the tests and it's working okay okay the only problem is that we are working currently with gp4 that might that may or may not not be a problem but that is the default language model that CI works on if you want to use a different language model I am going to show you right here how to change that default setting and so that you can use pretty much any language model that you want that is supported by

[1:04:39] langing okay so let's do that right now let's go to the browser to show you um the language models that I'm going to be using here okay so here we are at the Lang chain documentation page and the idea right here is that actually crei is built on top of Lang chain this means that actually Lang chain crei is going to be compatible with all of the Integrations for all of the language model providers that Lang chain has available this means that if you want to use gr with anthropic with Google or with open AI models for example it is going to work just like it would on Lang

[1:05:16] chain all you will have to do is change a single line of code I'm going to show you how to do that in a moment um just be sure that the language model that you choose supports function calling because of course since we're going to be working with agents agents require function calling in order to work okay so just whenever you're choosing your language model be sure to pick one that supports that for this example I am going to be showing you how to do this using anthropic models that is clot and I am going to be showing you how to use uh Gro okay all right so now it's time

[1:05:50] to get our API Keys um the first one that I'm going to be uh getting is GR Gro so in case you don't know what Gro is it is a service that allows you to serve open source models from their architecture and they have a very specific way of implementing their architecture which allows you to get inference or to get the completions from your language models much much faster than you would from a regular API so that is very useful and they also support function calling they have right here if you go to Gro Cloud create an account and go to documentation you can see the models that they have available

[1:06:29] in these case as you can see they have Lama 3 they have mixol and they also have whisper and Gemma okay in this example we're going to be using lth 3 and mixol however when you're using Grog be sure to be careful about the usage rates because they do have some rate limits right here as you can see they have some rate limits and their models are limited depending on on the region you are I think and also on your profile let's see right here settings so here you go if you don't have a very high limit this might not work in my case uh for my particular limits that I have I

[1:07:09] do find that I hit the rate limit pretty often so in my situation Gro is probably not the most useful but if you have a higher rate limit or if by the time you're watching this video they already have released their paid setting this is definitely a thing that you should take a look at because they they have very fast inference and it works pretty good so in order to do that let's just go to API Keys create a new API key and then we're going to copy it to ourv file and the second one that I wanted to show you is cloud of course in order to get an

[1:07:43] API key from anthropic which are the creators of CLA you're going to go to anthropic Doom API you click on get started now and it will take you to the console if you haven't created an account yet it's going to prompt you to create an account once you create an account you just have to go to get API keys and then you will be able to create your keys and copy them to ourv file okay so once we have our API Keys we can go back to vs code populate ourv file and start [Music]

[1:08:15] testing okay so we're back at our crew. py file where our uh automation lives here we have all of our agents and all of our tasks but if you're not working with this kind of setting if you're just working with the main abstractions yourself you can still uh do this all you have to do is that whever you're going to initialize your agent you're going to add an extra parameter and you're going to say llm equals and here is where you are going to initialize your Lang chain language model OKAY in this case if you might remember correctly we are going to be using anthropic and uh Gro so the first thing

[1:08:54] to do is actually to to install the Integrations from these language models okay so in order to install them we're going to have to go to our terminal right here and uh to make sure that I have my virtual environment installed and activate it and I am going to install integ integration um you can check what you have to install in the documentation itself for the language model that you're trying to implement in this case since I'm going to be using anthropic I can do pip install anthropic sorry pip install Lang chain Das anthropic or in my case since I am using poetry I'm going to do poetry

[1:09:30] at L chain anthropic oops length chain anthropic like that and it's probably going to take a little bit it's going to download it there you go it wasn't that long and once we're at it let's actually install Lang chain Croc remember that some implementations come I mean some third party Integrations come within the Lang chain Community package but in this case since they are pretty big Integrations they come within their own package and this is langin anthropic and langen Gro you will have to install whichever you are trying to use and you will find that in the documentation of course okay so now that we have that

[1:10:16] installed we're going to go Tov right here and it is right here that we're going to add the API keys that we just added and actually we're not going to be needing this this thing right here we can just paste our new API key so we have the open a API key we have the gro API key and the anthropic API key for these two particular language model providers be sure to use this environment variable name otherwise it is not going to work okay so Lang chain or crei and Lang chain are going to automatically look for this API sorry for this environment variable right here

[1:10:54] so so this environment variable name so you have to name this groc API key and anthropic API key if these are the models that you're going to be using I'm going to save this go back to crew. py and I'm just going to initialize my language model uh in order to do this actually I'm just going to initialize a new method right here I'm going to call it llm like this and I am just going to initialize my language model right here I'm going to say that llm equals and in the first case I'm going to be initializing a clot model so I'm going to do from Lang

[1:11:31] chain anthropic I'm going to import chat anthropic like this going to do the same with Gro so from L chain from L chain groc I'm going to import chat groc and let's just initialize both of them right here so I'm going to say llm equals chat anthropic and this one of course the a few parameters the first one is the model name and in my case I am going to be using Cloud Sonet which I found that it's actually the one that works the best here and one thing to keep in mind especially about the anthropic models with Lang chain is that by default they don't return a very large amount a very

[1:12:19] large number of tokens so here you're going to want to update that setting I'm going to say that Max tokens and I am going to set that to 49 to6 and this is going to allow my language model to return up to 4,096 tokens per call okay because by default I think it's around a th000 and it was generating I mean I I tested this before and it was generating truncated risk responses and the second one that I am going to be initializing is chat gr and just like up there we're going to be initializing it with a model name and in this case I'm going to be testing lameth

[1:12:57] 3 and that is actually all that we need I'm actually going to be initializing another one with mixol so let's get the mixt tral name right here right here just paste it and there we go and these are the three language models that I'm going to be testing clot sonit Lama 3 and mixl okay um to be clear I am using chat gr right here on the free version which is at the moment of recording the only one available but I mean yeah so this allows me to run my automations for free as long long as I'm willing to wait in order to not hit the rate limit per

[1:13:28] minute okay and of course I'm going to be using the chat anthropic which is a paid one and I have already added credits to my account for that one and I'm just going to return llm so the first one that I'm going to be testing is chat anthropic so let's save this and I'm going to initialize this one right here I'm going to say self LM going to say that we're going to be using the same one for this one right here and the same one for this one right here I mean just to be sure that you're not lost here all I am doing is I am calling the

[1:14:02] function that I initialized up here and I'm just going to be returning the instance of the language model that I just initialized okay so once that is done we can save this and now we should be able to run our automation using these new language models that we just imported great so in order to test this actually I wanted to show you something which is how to add logs to every single step of your automation okay because it is not always easy to see what your agents are doing and we're going to see a little bit later how to do monitoring and observability but for now I just

[1:14:39] want you to see exactly what each agent is uh producing uh by the end of each task okay only by the end of each task and that is going to allow us to see precisely what the editor did what the researcher did and what the final newsletter um file is okay so in order to do that we're going to add a new parameter right here to every single one of our tasks and we're going to call it output file like this and right here you're going just to name the name of the file or the location of the file where this output is going to be stored and it is

[1:15:19] going to produce a file by the end of this task which with all of the output of this task in this case I'm going to create a new folder right here called logs like that and I am going to also import daytime actually I'm going to import from daytime I'm going to import daytime and right here actually I am going to add an F string and inside here I am going to say that I want to store this in logs and I want my file to be named after the current datetime okay so current day time now and it's going to format it by day uh month sorry year

[1:16:00] month and day and actually I also want the hour because I'm probably going to be running several tests every uh on the same day so here I'm going to add hour minute and second and then I'm going to concatenate this with the name of the task which is research task I'm going to save it in an MD file in a markdown file and I'm going to do the same thing with my edit task I'm going to say that the output file is going to be in stored inside the logs directory the named with the current time of the execution and then I'm going to do exactly the same

[1:16:37] thing with the last one which is yes like this but this one is not going to be a markdown file but it's going to be a HTML file is you if you remember correctly okay great so once that is done every single one of our tasks is going to be stored in its own file in the logs directory now we can actually execute it and see how this actually looks like so I'm going to open the terminal I'm just going to do poetry run I'm going to run my automation called newslet gen again our deprecation warning the topic of our newsletter this time is going to be real estate market

[1:17:18] in in San Francisco and let's say a personal message for the newsletter hello everyone this is your Weekly Newsletter about real estate in SF enter and we have a quick error right here because newsletter gen llm takes zero positional arguments and one was given okay yeah I forgot to add here self like this okay so let's run this again and and yeah again it's going to be the real estate market in ASF and my message is going to be again hello everyone so let's see as you can see it is starting to thing it already started to query um the internet and it found a lot of

[1:18:17] information already it found some Bay Area Housing 20 billion very fun and going to be back once all of this run is complete but just to show you that everything is running correctly okay so all of the thinking has been finished by our agents all of their research and they actually communicated with each other a lot so that looked pretty good amazing so and also they solved their own errors and by the end they produced a final um a final newsletter right here let's see our logs and take a look at how that looks like so here we have as you can see that the logs were were saved as we as we asked

[1:19:04] them to the first one was the research task and let's see so here we have Story one story two Story 3 four five 6 7 8 and nine and here we have different titles so we have Goldman Sachs and B La have a summary and we have the URL and same thing for the Story 2 and as you you can see the language model is following my example of the of the stories that I of the format of the stories that I want so remember that I I mentioned before when I was writing the prompts that I wanted it to have this exact format um let me show you real

[1:19:44] quick so if we go to configuration and we go to tasks here you can see that I asked the research task on the expected output that they put that they write these stories like this story one title summary and URL and that is exactly what it did write right here great so the research task turned out all right let's see how the oops let's see how the edit task turned out and here we have just as well the title the summary why this is important a very quick um commentary by our editor and this Source like this great um oh we seem to have a problem apparently we

[1:20:31] only have I mean we lost four uh no we lost five stories from the researcher to the editor I suppose I mean I hope that it is because the editor found that those stories were repetitive or probably not relevant for our topic we're I mean later I'm going to show you how to do this with how to check the the logs and monitor or the thinking of your agents so that you know why they made certain decisions but for now let's just stick to what they returned to us and last but not least of course we have the final generation which was the goal of our entire automation let's see here we

[1:21:09] have the HTML file that was generated let's just serve it to see how it looks like and here it is so we see that it looks okay hello everyone this is your Weekly Newsletter about about real estate in a SEF and everything seems to be working correctly we have everything and the links are working correctly as well great so good job so the automation is working and we actually managed to make it work using anthropic clot Sonet and that turned out pretty great so now it is time to actually build a front end for this application so let's do that right now [Music]

[1:22:04] [Music]

[1:22:10] [Music]

---

About this transcript
Read from YouTube's own caption track and laid out by ViewRank AI (https://viewrankai.com).
ViewRank AI finds the videos already beating a creator's own average on Instagram, TikTok and YouTube Shorts, transcribes them from the audio itself in more than 60 languages, and turns what worked into new ideas and scripts.
Free transcript tools, no account needed: https://viewrankai.com/tools
How to read any video this way: https://viewrankai.com/llms.txt