The AiEdge Newsletter

Share this post

Deep Dive: How to Automate Writing Newsletters with LangChain, Stable Diffusion and DiaChat

newsletter.theaiedge.io
Newsletters

Deep Dive: How to Automate Writing Newsletters with LangChain, Stable Diffusion and DiaChat

Building Machine Learning Solutions

Damien Benveniste
Jun 1, 2023
∙ Paid
18
Share this post

Deep Dive: How to Automate Writing Newsletters with LangChain, Stable Diffusion and DiaChat

newsletter.theaiedge.io
3
Share

LangChain is certainly one of the most powerful technologies I came across recently. It allows to use Large Language Models as building blocks for software applications. Today I am exploring how much I can rely on ChatGPT and other generative AI tools to automate human-like activities such as writing a newsletter about the past week's news. We cover:

  • A newsletter fully generated by AI

  • Getting the top news

    • The NewsAPI API

    • Selecting the top news with ChatGPT

  • Summarizing the news

    • Extracting the news text

    • Summarizing the text with LangChain

  • Generating images with Stable Diffusion

    • The Stable Diffusion API

    • Generating images for the summaries

  • Illustrating with a diagram using DiaChat

  • Generating a title and abstract

    • The abstract

    • The title


A newsletter fully generated by AI

I have been thinking about writing newsletters to capture the latest news about Machine Learning for a while now, but I was having difficulties finding the time to do it. Therefore, I created a software that automates this process! Here is an example of such a newsletter:

Newsletters

The Future of AI: From Generative Podcasts to Climate Change Solutions

Damien Benveniste
·
May 31
The Future of AI: From Generative Podcasts to Climate Change Solutions

Read full story

I will refine the process over time, but I am curious, would you be interested if I sent such a newsletter every week?

Loading...

Now let's get into how to build such an automation! Make sure to install the necessary libraries and set up your OpenAI API key on their platform:

pip install langchain openai newsapi-python beautifulsoup4

Getting the top news

The NewsAPI API

I want to highlight the most interesting news about Machine Learning in the past week. I use NewsAPI to access the news. Login and get your API key. You can find documentation of the Python client library here:

Let’s get all the news about “Artificial Intelligence“ from the past week:

from newsapi import NewsApiClient

newsapi = NewsApiClient(api_key=NEWS_API_KEY)

everything = newsapi.get_everything(
    q='artificial intelligence',
    from_param='2023-05-22',
    to='2023-05-29',
    sort_by='relevancy',
    language='en'
)

everything['articles'][0]

The free plan only provides a maximum of 100 articles but it is enough to find interesting articles. I could have sorted the results by “popularity“ as well, but I was afraid of getting too many irrelevant articles.

Selecting the top news

100 articles is too many to highlight in a newsletter, so I will select only 10 of them. Because I am lazy, I will ask ChatGPT to find the most appropriate ones for me! I use LangChain to do so.

I am creating the following prompt template so I can rerun the code multiple times with different arguments. I am asking it to extract the 10 most important news about machine learning and return it as a Python list. This is so I can directly use the results in a script:

from langchain import PromptTemplate

top_news_template = """
extract from the following list the 10 most important news about machine learning. Return the answer as a python list. 

For example: ['text 1', 'text 2', 'text 3']

{list}
"""

TOP_NEWS_PROMPT = PromptTemplate(
    input_variables=['list'],
    template=top_news_template,
)

In the above prompt, I had to show it an example (few-shot prompting) as it sometimes failed to return the result as a Python list. I can now create a simple chain with ChatGPT as the model:

from langchain.chains import LLMChain
from langchain.chat_models import ChatOpenAI

llm = ChatOpenAI()
chain = LLMChain(
    llm=llm,
    prompt=TOP_NEWS_PROMPT
)

For the list, I will use article titles. One can argue it is not the most effective way to select relevant news but it is a first pass at getting us there

title_list = '\n'.join([
    article['title'] 
    for article in everything['articles']
])

title_list 

Let’s run that and convert the result into a Python list so we can use it later:

import ast

# We send the prompt to ChatGPT
top_str = chain.run(title_list)

# And we convert the result as a python list
top_list = ast.literal_eval(top_str)
top_list

Let’s match those titles with the related URLs:

top_news = [
    {
        'title': a['title'],
        'url': a['url']
    } 
    for a in everything['articles']
    if a['title'] in top_list
]

top_news

Obviously I am giving a lot of confidence in ChatGPT’s ability to return the data in the right format. If it was a more formal software development, I would be less relaxed about it.

Summarizing the news

Extracting the news text

I would like to provide a summary of each of the selected articles. I will extract the text of the related articles and summarize them. We can easily extract the text using the URL using Beautiful Soup:

Keep reading with a 7-day free trial

Subscribe to

The AiEdge Newsletter
to keep reading this post and get 7 days of free access to the full post archives.

Already a paid subscriber? Sign in
Previous
Next
© 2023 AiEdge
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing