Deep Dive: How to Automate Writing Newsletters with LangChain, Stable Diffusion and DiaChat
Building Machine Learning Solutions
LangChain is certainly one of the most powerful technologies I came across recently. It allows to use Large Language Models as building blocks for software applications. Today I am exploring how much I can rely on ChatGPT and other generative AI tools to automate human-like activities such as writing a newsletter about the past week's news. We cover:
A newsletter fully generated by AI
Getting the top news
The NewsAPI API
Selecting the top news with ChatGPT
Summarizing the news
Extracting the news text
Summarizing the text with LangChain
Generating images with Stable Diffusion
The Stable Diffusion API
Generating images for the summaries
Illustrating with a diagram using DiaChat
Generating a title and abstract
The abstract
The title
A newsletter fully generated by AI
I have been thinking about writing newsletters to capture the latest news about Machine Learning for a while now, but I was having difficulties finding the time to do it. Therefore, I created a software that automates this process! Here is an example of such a newsletter:
I will refine the process over time, but I am curious, would you be interested if I sent such a newsletter every week?
Now let's get into how to build such an automation! Make sure to install the necessary libraries and set up your OpenAI API key on their platform:
pip install langchain openai newsapi-python beautifulsoup4
Getting the top news
The NewsAPI API
I want to highlight the most interesting news about Machine Learning in the past week. I use NewsAPI to access the news. Login and get your API key. You can find documentation of the Python client library here:
Let’s get all the news about “Artificial Intelligence“ from the past week:
from newsapi import NewsApiClient
newsapi = NewsApiClient(api_key=NEWS_API_KEY)
everything = newsapi.get_everything(
q='artificial intelligence',
from_param='2023-05-22',
to='2023-05-29',
sort_by='relevancy',
language='en'
)
everything['articles'][0]
The free plan only provides a maximum of 100 articles but it is enough to find interesting articles. I could have sorted the results by “popularity“ as well, but I was afraid of getting too many irrelevant articles.
Selecting the top news
100 articles is too many to highlight in a newsletter, so I will select only 10 of them. Because I am lazy, I will ask ChatGPT to find the most appropriate ones for me! I use LangChain to do so.
I am creating the following prompt template so I can rerun the code multiple times with different arguments. I am asking it to extract the 10 most important news about machine learning and return it as a Python list. This is so I can directly use the results in a script:
from langchain import PromptTemplate
top_news_template = """
extract from the following list the 10 most important news about machine learning. Return the answer as a python list.
For example: ['text 1', 'text 2', 'text 3']
{list}
"""
TOP_NEWS_PROMPT = PromptTemplate(
input_variables=['list'],
template=top_news_template,
)
In the above prompt, I had to show it an example (few-shot prompting) as it sometimes failed to return the result as a Python list. I can now create a simple chain with ChatGPT as the model:
from langchain.chains import LLMChain
from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI()
chain = LLMChain(
llm=llm,
prompt=TOP_NEWS_PROMPT
)
For the list, I will use article titles. One can argue it is not the most effective way to select relevant news but it is a first pass at getting us there
title_list = '\n'.join([
article['title']
for article in everything['articles']
])
title_list
Let’s run that and convert the result into a Python list so we can use it later:
import ast
# We send the prompt to ChatGPT
top_str = chain.run(title_list)
# And we convert the result as a python list
top_list = ast.literal_eval(top_str)
top_list
Let’s match those titles with the related URLs:
top_news = [
{
'title': a['title'],
'url': a['url']
}
for a in everything['articles']
if a['title'] in top_list
]
top_news
Obviously I am giving a lot of confidence in ChatGPT’s ability to return the data in the right format. If it was a more formal software development, I would be less relaxed about it.
Summarizing the news
Extracting the news text
I would like to provide a summary of each of the selected articles. I will extract the text of the related articles and summarize them. We can easily extract the text using the URL using Beautiful Soup:
Keep reading with a 7-day free trial
Subscribe to