Deep Dive: How to Automate Writing Newsletters with LangChain, Stable Diffusion and DiaChat

Building Machine Learning Solutions

LangChain is certainly one of the most powerful technologies I came across recently. It allows to use Large Language Models as building blocks for software applications. Today I am exploring how much I can rely on ChatGPT and other generative AI tools to automate human-like activities such as writing a newsletter about the past week's news. We cover:

• A newsletter fully generated by AI

• Getting the top news

  • The NewsAPI API

  • Selecting the top news with ChatGPT

• Summarizing the news

  • Extracting the news text

  • Summarizing the text with LangChain

• Generating images with Stable Diffusion

  • The Stable Diffusion API

  • Generating images for the summaries

• Illustrating with a diagram using DiaChat

• Generating a title and abstract

  • The abstract

  • The title

---

A newsletter fully generated by AI

I have been thinking about writing newsletters to capture the latest news about Machine Learning for a while now, but I was having difficulties finding the time to do it. Therefore, I created a software that automates this process! Here is an example of such a newsletter:

I will refine the process over time, but I am curious, would you be interested if I sent such a newsletter every week?

Now let's get into how to build such an automation! Make sure to install the necessary libraries and set up your OpenAI API key on their platform:

pip install langchain openai newsapi-python beautifulsoup4

Getting the top news

The NewsAPI API

I want to highlight the most interesting news about Machine Learning in the past week. I use NewsAPI to access the news. Login and get your API key. You can find documentation of the Python client library here:

Let’s get all the news about “Artificial Intelligence“ from the past week:

from newsapi import NewsApiClient

newsapi = NewsApiClient(api_key=NEWS_API_KEY)

everything = newsapi.get_everything(
    q='artificial intelligence',
    from_param='2023-05-22',
    to='2023-05-29',
    sort_by='relevancy',
    language='en'
)

everything['articles'][0]

The free plan only provides a maximum of 100 articles but it is enough to find interesting articles. I could have sorted the results by “popularity“ as well, but I was afraid of getting too many irrelevant articles.

Selecting the top news

100 articles is too many to highlight in a newsletter, so I will select only 10 of them. Because I am lazy, I will ask ChatGPT to find the most appropriate ones for me! I use LangChain to do so.

I am creating the following prompt template so I can rerun the code multiple times with different arguments. I am asking it to extract the 10 most important news about machine learning and return it as a Python list. This is so I can directly use the results in a script:

from langchain import PromptTemplate

top_news_template = """
extract from the following list the 10 most important news about machine learning. Return the answer as a python list. 

For example: ['text 1', 'text 2', 'text 3']

{list}
"""

TOP_NEWS_PROMPT = PromptTemplate(
    input_variables=['list'],
    template=top_news_template,
)

In the above prompt, I had to show it an example (few-shot prompting) as it sometimes failed to return the result as a Python list. I can now create a simple chain with ChatGPT as the model:

from langchain.chains import LLMChain
from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI()
chain = LLMChain(
    llm=llm,
    prompt=TOP_NEWS_PROMPT
)

For the list, I will use article titles. One can argue it is not the most effective way to select relevant news but it is a first pass at getting us there

title_list = '\n'.join([
    article['title'] 
    for article in everything['articles']
])

title_list 

Let’s run that and convert the result into a Python list so we can use it later:

import ast

# We send the prompt to ChatGPT
top_str = chain.run(title_list)

# And we convert the result as a python list
top_list = ast.literal_eval(top_str)
top_list

Let’s match those titles with the related URLs:

top_news = [
    {
        'title': a['title'],
        'url': a['url']
    } 
    for a in everything['articles']
    if a['title'] in top_list
]

top_news

Obviously I am giving a lot of confidence in ChatGPT’s ability to return the data in the right format. If it was a more formal software development, I would be less relaxed about it.

Summarizing the news

Extracting the news text

I would like to provide a summary of each of the selected articles. I will extract the text of the related articles and summarize them. We can easily extract the text using the URL using Beautiful Soup: