README.md

---
title: TweetGPT
emoji: 🚀
colorFrom: purple
colorTo: blue
sdk: streamlit
sdk_version: 1.34.0
app_file: app.py
pinned: true
license: apache-2.0
---

# Political Parrots: GPT’s Take on Bundestag Tweets

**Tim Michalow (qo27leja), Ian Fischer (uf28alic), Tobias Stirner (zo94suqa), Jonathan Franke (zu98wibu)**

## 1 Introduction

### Motivation

The use of social media in political communication has surged in recent years, with platforms like Twitter becoming pivotal in shaping public discourse. This trend was notably exemplified during Donald Trump’s election campaign, which underscored Twitter’s significant influence on political narratives and engagement. Recognizing the growing importance of social media in politics, this study focuses on analyzing the communication strategies of German political parties on Twitter.

### Define your research question

How effectively can advanced NLP techniques, particularly transformer-based models such as GPT-2, replicate and simulate the rhetorical styles and sentiment patterns of tweets posted by German political parties on Twitter?

### How is this document structured

This document is structured as follows:
1. Introduction
2. Related Work
3. Methodology
4. Results
5. Discussion
6. Conclusion

## 2 Related Work

### NLP and Social Media

The field of natural language processing (NLP) has significantly advanced with the introduction of transformer-based models such as GPT-2 and BERT. Radford et al. (2019) in "Language Models are Unsupervised Multitask Learners" highlight the capabilities of GPT-2 in generating coherent and contextually relevant text. This study underscores the potential of GPT-2 to replicate the complex language patterns found in political communication on social media platforms like Twitter.

### Political Communication on Social Media

Research on political communication emphasizes the strategic use of social media by political entities to engage with the public and influence opinions. Enli and Skogerbø (2013) in "Personalized campaigns in party-centred politics" explore how political parties leverage social media for personalized campaigns to reach voters effectively. Jungherr (2016) in "Analyzing Political Communication with Digital Trace Data" provides methodologies for studying political behavior on social media, which inform our data collection and analysis strategies.

### Sentiment Analysis and Rhetorical Style in Tweets

Understanding the sentiment and rhetorical style in tweets is crucial for analyzing political communication. Balahur et al. (2013) in "Sentiment Analysis in Twitter" examine various techniques for determining sentiment in short texts, which is pertinent to our evaluation metrics. Hart and Lind (2011) in "Political Rhetoric and its Discontents" showcase the importance of rhetorical analysis in political discourse, guiding our approach to replicating the distinctive styles of different political parties.

### Applications of GPT-2 in Text Generation

The practical applications of GPT-2 for text generation have been demonstrated in various domains. Brown et al. (2020) in "Language Models are Few-Shot Learners" show the model's ability to generate human-like text with minimal fine-tuning, validating our choice of GPT-2 for generating political tweets. Furthermore, Wu et al. (2019) in "GPT-2 as a Knowledge Engine" illustrate the model's capacity to generate relevant and coherent content, supporting our deployment strategy.

### Implications of AI in Political Communication

The ethical implications of using AI in political communication are a significant concern. Floridi et al. (2018) in "AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations" discuss the ethical challenges posed by AI, such as bias and misinformation, which are relevant to our study. Additionally, the "Automating Society Report" by AlgorithmWatch (2020) provides insights into the societal impacts of automated decision-making systems, including those used in political communication.

By integrating insights from these studies, our project aims to build upon the existing body of knowledge and contribute to the field of political communication analysis through advanced NLP techniques.

## 3 Methodology

### 3.1 General Methodology

Our methodology involves several key steps to achieve our project goals:

1. **Data Collection**: We collected tweets from the official Twitter accounts of German political parties.
2. **Data Preparation**: We cleaned and prepared the dataset for analysis, including filtering out retweets and irrelevant information.
3. **Model Training**: We fine-tuned a GPT-2 model on our dataset to generate tweets that mimic the style of each political party.
4. **Evaluation**: We evaluated the model’s performance using various metrics to ensure the generated tweets were accurate and relevant.
5. **Deployment**: We developed a Streamlit application to deploy the trained model and allow users to generate tweets interactively.

### 3.2 Data Understanding and Preparation

#### 3.2.1 Merging JSON Lines Files

We merged multiple JSON Lines files containing tweet data into a single dataset to streamline our analysis process.

#### 3.2.2 Filtering Retweets

To focus on original content, we filtered out retweets from the dataset. This ensured that our analysis was based solely on the tweets created by the political parties themselves.

#### 3.2.3 Cleaning Tweet Text

We cleaned the tweet text by removing special characters, URLs, and other irrelevant information. This preprocessing step was crucial to ensure the quality of the data used for model training.

### 3.3 Modeling and Evaluation

#### 3.3.1 Model and Tokenizer

We used a fine-tuned GPT-2 model for tweet generation. The tokenizer was configured to handle the specific nuances of the German language and political terminology.

#### 3.3.2 Parameters

The model was fine-tuned with the following parameters:

Model: GPT-2
Training epochs: 3
Batch size: 6
Learning rate: 2e-4
Warmup steps: 100
Epsilon: 1e-8
Evaluation strategy: Evaluate after each epoch
Save strategy: Save the model after each epoch
Seed: 38

#### 3.3.3 Training

The training process involved feeding the cleaned and prepared dataset into the GPT-2 model. We used a combination of supervised learning techniques to fine-tune the model effectively.
We trained the model using the Hugging Face Trainer, which takes the parameters as
input. We opted for this because it is optimized for transformer and also comes from
the same framework. During training, we used the WANDB API to track the training
of each model of the respective parties and obtain metrics. We ran the training of the
models via Kaggle, as Kaggle provides two T4 GPUs and so we got good hardware
without paying anything. Another advantage was that we could run the training via
CUDA. The training took between 2 and 10 hours, depending on the number of tweets
from each party. We will go into this in more detail in the evaluation.

#### 3.3.4 Generation and Deployment

After training, we used the model to generate tweets that mimic the styles of different political parties. The deployment was handled through a Streamlit application, providing an interactive platform for users to generate and analyze tweets.

## 4 Results

### 4.1 Artifacts

The main artifacts produced from this project include:
- A fine-tuned GPT-2 model for tweet generation
- A cleaned and prepared dataset of German political tweets
- A Streamlit application for tweet generation and analysis

### 4.2 Libraries and Tools

We used several libraries and tools, including:
- TensorFlow and PyTorch for model training
- Hugging Face Transformers for model fine-tuning
- Pandas and NumPy for data processing
- Streamlit for application deployment

### 4.3 Concept of the App

Our Streamlit application allows users to select a political party and generate tweets that reflect the party's communication style. The app provides an interactive and user-friendly interface for exploring the generated tweets.

### 4.4 Results on Unseen Data

The model performed well on unseen data, generating tweets that were coherent and stylistically similar to those of the respective political parties. The generated tweets were evaluated based on their relevance, sentiment, and rhetorical style.

### 4.5 Visualizations from WandB

We used Weights and Biases (WandB) for tracking and visualizing our machine learning experiments. The following screenshots provide insights into the training process and performance metrics of our model:

##### 4.5.1 Evaluation Metrics

- **Steps per Second**
![Steps per Second](./WandB_Training_Evaluation/eval_stepspersecond.png)

- **Runtime**
![Runtime](./WandB_Training_Evaluation/eval_runtime.png)

- **Samples per Second**
![Samples per Second](./WandB_Training_Evaluation/eval_samplespersecond.png)

- **Loss**
![Loss](./WandB_Training_Evaluation/eval_loss.png)

##### 4.5.2 Training Metrics

- **Gradient Norm**
![Gradient Norm](./WandB_Training_Evaluation/train_gradnorm.png)

- **Global Step**
![Global Step](./WandB_Training_Evaluation/train_globalstep.png)

- **Loss**
![Training Loss](./WandB_Training_Evaluation/train_loss.png)

- **Learning Rate**
![Learning Rate](./WandB_Training_Evaluation/train_learningrate.png)

- **Epoch**
![Epoch](./WandB_Training_Evaluation/train_epoch.png)

## 5 Discussion

### 5.1 Results/Artifacts/App

The generated tweets successfully mimicked the communication styles of different German political parties. The Streamlit application provided an intuitive platform for users to interact with the model and generate tweets.

### 5.2 Limitations

Several limitations were identified during the project:
- The dataset was limited in size, which may have affected the model's ability to generalize.
- GPU availability in Colab and Kaggle was a constraint, limiting the extent of model fine-tuning.
- The app's functionality was limited to tweet generation and did not include advanced features like real-time sentiment analysis.

### 5.3 Ethical Perspective

#### Dangers of the Application

The use of AI-generated content in political communication can lead to ethical concerns, such as the potential for discrimination through biased models or the spread of misinformation.

#### Transparency

Ensuring transparency in the model's training and deployment processes is crucial to maintain public trust and accountability.

#### Effects on Climate Change

The computational resources required for training large models have a significant carbon footprint, contributing to climate change.

Possible sources for further ethical considerations:
- "Automating Society Report"
- Relevant publications on the ethics of AI and social media

### 5.4 Further Research

Future research could explore:
- Expanding the dataset to include more tweets and other forms of social media communication.
- Incorporating real-time sentiment analysis and other advanced features into the application.
- Investigating the long-term impacts of AI-generated content on public opinion and political engagement.

## 6 Conclusion

In this project, we analyzed the communication strategies of German political parties on Twitter using a fine-tuned GPT-2 model. Our results demonstrate the potential of NLP techniques in political communication analysis. Future research could build on these findings to explore more advanced applications and address the ethical implications of AI in social media.