changing to mastodon fetcher

README.md

@@ -12,17 +12,17 @@ pinned: false
 # What would mother say?
 
 This app includes a Haystack agent with access to 2 tools:
-- `TweetRetriever`: Useful for when you need to retrive the latest tweets from a username to get an understanding of their style
+- `MastodonRetriever`: Useful for when you need to retrive the latest posts from a username to get an understanding of their style
 - `WebSearch`: Useful for when you need to research the latest about a new topic
 
-We build an Agent that aims to first understand the style in which a username tweets. Then, it uses the WebSearch tool to gain knowledge on a topic that the LLM may not have info on, to generate a tweet in the users style about that topic.
+We build an Agent that aims to first understand the style in which a username posts. Then, it uses the WebSearch tool to gain knowledge on a topic that the LLM may not have info on, to generate a post in the users style about that topic.
 ### Try it out on [🤗 Spaces](https://huggingface.co/spaces/Tuana/what-would-mother-say)
 
 ##### A showcase of a Haystack Agent with a custom `TwitterRetriever` Node and a `WebQAPipeline` as tools.
 
 **Custom Haystack Node**
 
-This repo contains a streamlit application that given a query about what a certain twitter username would post on a given topic, generates a tweet in their style (or tries to). It does so by using a custom Haystack node I've built called the [`TweetRetriever`](/custom_nodes/tweet_retriever.py)
+This repo contains a streamlit application that given a query about what a certain twitter username would post on a given topic, generates a post in their style (or tries to). It does so by using a custom Haystack node I've built called the [`MastodonFetcher`](https://haystack.deepset.ai/integrations/mastodon-fetcher)
 
 **Custom PromptTemplates**
 

app.py

@@ -6,29 +6,30 @@ import streamlit as st
 
 from utils.haystack import run_agent, start_haystack
 from utils.ui import reset_results, set_initial_state, sidebar
-from utils.config import TWITTER_BEARER_TOKEN, OPENAI_API_KEY, SERPER_KEY
+from utils.config import OPENAI_API_KEY, SERPER_KEY
 
 set_initial_state()
 
 sidebar()
 
-st.write("# 👩 What would they have tweeted about this?")
+st.write("# 👩 What would they have posted about this?")
+st.write("### This app had to be changed from searching Twitter posts to searching Mastodon posts due to API changes")
 st.write("## About this App")
-st.write("This app, built with [Haystack](https://github.com/deepset-ai/haystack#readme) uses an [Agent](https://docs.haystack.deepset.ai/docs/agent) (with GPT-4) with a `WebSearch` and `TwitterRetriever` tool")
-st.write("The `TwitterRetriever` is set to retrieve the latest 15 tweets by the given twitter username to construct an understanding of their style of tweeting")
+st.write("This app, built with [Haystack](https://github.com/deepset-ai/haystack#readme) uses an [Agent](https://docs.haystack.deepset.ai/docs/agent) (with GPT-4) with a `WebSearch` and `MastodonRetriever` tool")
+st.write("The `MastodonRetriever` is set to retrieve the latest 20 mastodon posts by the given Mastodon username to construct an understanding of their style of posting")
 st.write("### Instructions")
-st.write("For best results, please provide a Twitter username as it appears on twitter. E.g.: [dog_feelings](https://twitter.com/dog_feelings) 🐶")
+st.write("For best results, please provide a Mastodon username as it appears on twitter. E.g.: [[email protected]](https://mastodon.social/@ivepetthatdog) 🐶")
 st.write("### Example")
-st.image("./images/example.png")
+st.image("./images/mastodon_example.png")
 
 # if st.session_state.get("OPENAI_API_KEY") and st.session_state.get("SERPER_KEY"):
 
-agent = start_haystack(openai_key=OPENAI_API_KEY, twitter_bearer=TWITTER_BEARER_TOKEN, serper_key=SERPER_KEY)
+agent = start_haystack(openai_key=OPENAI_API_KEY, serper_key=SERPER_KEY)
 # st.session_state["api_keys_configured"] = True
 
 # Search bar
-question = st.text_input("Example: If the twitter account tuanacelik were to write a tweet in their style about climate change, what would it be?", on_change=reset_results) 
-run_pressed = st.button("Generate tweet")
+question = st.text_input("Example: If the twitter account tuanacelik were to write a post in their style about climate change, what would it be?", on_change=reset_results) 
+run_pressed = st.button("Generate post")
 # else:
 #     st.write("Please provide your OpenAI and SerperDev Keys to start using the application")
 #     st.write("If you are using a smaller screen, open the sidebar from the top left to provide your OpenAI Key 🙌")

prompts/{twitter_agent.yaml → mastodon_agent.yaml}

@@ -1,12 +1,12 @@
-description: An experimental Agent prompt for tweet creations based on twitter voice or last tweets
+description: An experimental Agent prompt for post creations based on Mastodon voice or last poosts
 meta:
   authors:
   - Tuana
-name: Tuana/twitter-agent
+name: Tuana/mastodon-agent
 tags:
 - agent
 text: | 
-    You are a helpful and knowledgeable agent. To achieve your goal of generating new tweets for twitter users, you have access to the following tools:
+    You are a helpful and knowledgeable agent. To achieve your goal of generating new posts for mastodon users, you have access to the following tools:
     {tool_names_with_descriptions}
     To answer questions, you'll need to go through multiple steps involving step-by-step thinking and selecting appropriate tools and their inputs; tools 
     will respond with observations. When you are ready for a final answer, respond with the `Final Answer:`
@@ -18,7 +18,7 @@ text: |
     Observation: the tool will respond with the result
     ...
 
-    Final Answer: the final answer to the question, make it a catchy tweet with the users voice
+    Final Answer: the final answer to the question, make it a catchy post with the users voice
     Thought, Tool, Tool Input, and Observation steps can be repeated multiple times, but sometimes we can find an answer in the first pass
     ---
 

prompts/{twitter_voice.yaml → mastodon_voice.yaml}

@@ -1,30 +1,30 @@
-description: An experimental few-shot prompt to syntehsize the 'twitter voice' of a uesrname
+description: An experimental few-shot prompt to syntehsize the 'mastodon voice' of a uesrname
 meta:
   authors:
   - Tuana
-name: Tuana/twitter-voice
+name: Tuana/mastodon-voice
 tags:
 - summarization
 text: | 
-    You will be given a twitter stream belonging to a specific profile. Answer with a summary of what they've lately been tweeting about and in what languages.
-    You may go into some detail about what topics they tend to like tweeting about. Please also mention their overall tone, for example: positive,
+    You will be given a Mastodon stream belonging to a specific profile. Answer with a summary of what they've lately been posting about and in what languages.
+    You may go into some detail about what topics they tend to like posting about. Please also mention their overall tone, for example: positive,
     negative, political, sarcastic or something else.
     Examples: 
     
-    Twitter stream: RT @deepset_ai: Come join our Haystack server for our first Discord event tomorrow, a deepset AMA session with @rusic_milos @malte_pietsch…
+    Mastodon stream: RT @deepset_ai: Come join our Haystack server for our first Discord event tomorrow, a deepset AMA session with @rusic_milos @malte_pietsch…
     RT @deepset_ai: Join us for a chat! On Thursday 25th we are hosting a 'deepset - Ask Me Anything' session on our brand new Discord. Come…
     RT @deepset_ai: Curious about how you can use @OpenAI GPT3 in a Haystack pipeline? This week we released Haystack 1.7 with which we introdu…
     RT @tuanacelik: So many updates from @deepset_ai today! 
     
-    Summary: This user has lately been retweeting tweets fomr @deepset_ai. The topics of the tweets have been around the Haystack community, NLP and GPT. They've
+    Summary: This user has lately been reposting posts fomr @deepset_ai. The topics of the posts have been around the Haystack community, NLP and GPT. They've
     been posting in English, and have had a positive, informative tone.
     
-    Twitter stream: I've directed my team to set sharper rules on how we deal with unidentified objects.\n\nWe will inventory, improve ca… 
+    Mastodon stream: I've directed my team to set sharper rules on how we deal with unidentified objects.\n\nWe will inventory, improve ca… 
     the incursion by China’s high-altitude balloon, we enhanced radar to pick up slower objects.\n \nBy doing so, w…
     I gave an update on the United States’ response to recent aerial objects. 
-    Summary: This user has lately been tweeting about having sharper rules to deal with unidentified objects and an incursuin by China's high-altitude
-    baloon. Their tweets have mostly been neutral but determined in tone. They mostly post in English.
-    Twitter stream: {tweets} 
+    Summary: This user has lately been posting about having sharper rules to deal with unidentified objects and an incursuin by China's high-altitude
+    baloon. Their posts have mostly been neutral but determined in tone. They mostly post in English.
+    Mastodon stream: {posts} 
     
     Summary: 
 version: 0.1.1

requirements.txt

@@ -1,5 +1,7 @@
-farm-haystack[inference]==1.18.0
+safetensors==0.3.3.post1
+farm-haystack[inference]==1.20.0
 streamlit==1.21.0
 markdown
 st-annotated-text
-python-dotenv
+python-dotenv
+mastodon-fetcher-haystack

utils/haystack.py

@@ -1,14 +1,13 @@
 import streamlit as st
-import requests
-from custom_nodes.tweet_retriever import TwitterRetriever
+from mastodon_fetcher_haystack import MastodonFetcher
 from haystack.agents import Agent, Tool
 from haystack.nodes import PromptNode, PromptTemplate, WebRetriever
 from haystack.pipelines import WebQAPipeline
 
 
-def start_haystack(openai_key, twitter_bearer, serper_key):
+def start_haystack(openai_key, serper_key):
     prompt_node = PromptNode(
-        "text-davinci-003",
+        "gpt-4",
         default_prompt_template=PromptTemplate(prompt="./prompts/lfqa.yaml"),
         api_key=openai_key,
         max_length=256,
@@ -17,17 +16,17 @@ def start_haystack(openai_key, twitter_bearer, serper_key):
     web_retriever = WebRetriever(api_key=serper_key, top_search_results=2, mode="preprocessed_documents")
     web_pipeline = WebQAPipeline(retriever=web_retriever, prompt_node=prompt_node)
 
-    twitter_retriver = TwitterRetriever(bearer_token=twitter_bearer, last_k_tweets=15)
+    mastodon_retriver = MastodonFetcher(last_k_posts=20)
 
     pn = PromptNode(model_name_or_path="gpt-4", api_key=openai_key, stop_words=["Observation:"], max_length=400)
-    agent = Agent(prompt_node=pn, prompt_template="./prompts/twitter_agent.yaml")
-
-    tweet_retriver_tool = Tool(name="TwitterRetriever", pipeline_or_node=twitter_retriver, 
-                               description="Useful for when you need to retrieve the latest tweets from a username to get an understanding of their style", 
-                               output_variable="results")
+    agent = Agent(prompt_node=pn, prompt_template="./prompts/mastodon_agent.yaml")
+    
+    mastodon_retriver_tool = Tool(name="MastodonRetriever", pipeline_or_node=mastodon_retriver, 
+                               description="Useful for when you need to retrieve the latest posts from a username to get an understanding of their style", 
+                               output_variable="documents")
     web_tool = Tool(name="WebSearch", pipeline_or_node=web_pipeline, description="Useful for when you need to research the latest about a new topic")
 
-    agent.add_tool(tweet_retriver_tool)
+    agent.add_tool(mastodon_retriver_tool)
     agent.add_tool(web_tool)
     
 
@@ -37,8 +36,5 @@ def start_haystack(openai_key, twitter_bearer, serper_key):
 
 @st.cache_data(show_spinner=True)
 def run_agent(_agent, question):
-    try:
-        result = _agent.run(question)
-    except Exception as e:
-        result = ["Life isn't ideal sometimes and this Agent fails at doing a good job.. Maybe try another query..."]
+    result = _agent.run(question)
     return result

utils/ui.py

@@ -31,7 +31,7 @@ def sidebar():
         st.markdown(
             "## How to use\n"
             # "1. Enter your [OpenAI API](https://platform.openai.com/account/api-keys) and [SerperDev API](https://serper.dev/) keys below\n"
-            "1. Enter a query that includes a twitter username and be descriptive about wanting a tweet as a result.\n"
+            "1. Enter a query that includes a Mastodon username and be descriptive about wanting a post as a result.\n"
             "2. Enjoy 🤗\n"
         )
 
@@ -62,7 +62,7 @@ def sidebar():
             "## How this works\n"
             "This app was built with [Haystack](https://haystack.deepset.ai) using the"
             " [`Agent`](https://docs.haystack.deepset.ai/docs/agent) custom [`PromptTemplates`](https://docs.haystack.deepset.ai/docs/prompt_node#templates)\n\n"
-            "as well as a custom `TweetRetriever` node\n"
+            "as well as a custom `MastodonFetcher` node\n"
             " The source code is also on [GitHub](https://github.com/TuanaCelik/what-would-mother-say)"
             " with instructions to run locally.\n"
             "You can see how the `Agent` was set up [here](https://github.com/TuanaCelik/what-would-mother-say/blob/main/utils/haystack.py)")