|
--- |
|
license: apache-2.0 |
|
metrics: |
|
- accuracy |
|
base_model: |
|
- meta-llama/Llama-3.1-8B-Instruct |
|
datasets: |
|
- stanfordnlp/nnetnav-wa |
|
--- |
|
|
|
# Model Card for Llama8b-NNetNav-WA |
|
|
|
<!-- Provide a quick summary of what the model is/does. [Optional] --> |
|
LLama8b-NNetNav-WA is a [LLama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) model that is instruct-tuned with [NNetNav-WA](https://huggingface.co/datasets/stanfordnlp/nnetnav-wa) data collected via unsupervised exploration on [WebArena](http://webarena.dev) websites, with a larger [LLama-3.1-70B](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) model. |
|
|
|
More details about this model can be found in our paper: [NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild](https://arxiv.org/abs/2410.02907). |
|
|
|
|
|
## Table of Contents |
|
|
|
- [Model Card for Llama8b-NNetNav-WA](#model-card-for--model_id-) |
|
- [Table of Contents](#table-of-contents) |
|
- [Model Details](#model-details) |
|
- [Results on Web-Agent Benchmarks](#results-on-benchmarks) |
|
- [Bias, Risks, and Limitations](#bias-risks-and-limitations) |
|
- [Training Details](#training-details) |
|
- [Training Data](#training-data) |
|
- [Training Procedure](#training-procedure) |
|
- [Environmental Impact](#environmental-impact) |
|
- [Technical Specifications](#technical-specifications) |
|
- [Hardware](#hardware) |
|
- [Software](#software) |
|
- [Model Card Authors [optional]](#model-card-authors-optional) |
|
- [Model Card Contact](#model-card-contact) |
|
- [How to Get Started with the Model](#how-to-get-started-with-the-model) |
|
|
|
## Model Details |
|
This model is intended to be used as a **web-agent** i.e. given an instruction such as _Upvote the post by user smurty123 on subreddit r/LocalLLaMA_, and a web-url _reddit.com_, the model can perform the task by executing a sequence of actions. |
|
|
|
<!-- Provide a longer summary of what this model is/does. --> |
|
The action space of the model is as follows: |
|
```plaintext |
|
Page Operation Actions: |
|
`click [id]`: This action clicks on an element with a specific id on the webpage. |
|
`type [id] [content] [press_enter_after=0|1]`: Use this to type the content into the field with id. By default, the "Enter" key is pressed after typing unless press_enter_after is set to 0. |
|
`hover [id]`: Hover over an element with id. |
|
`press [key_comb]`: Simulates the pressing of a key combination on the keyboard (e.g., Ctrl+v). |
|
`scroll [down|up]`: Scroll the page up or down. |
|
|
|
Tab Management Actions: |
|
`new_tab`: Open a new, empty browser tab. |
|
`tab_focus [tab_index]`: Switch the browser's focus to a specific tab using its index. |
|
`close_tab`: Close the currently active tab. |
|
|
|
URL Navigation Actions: |
|
`goto [url]`: Navigate to a specific URL. |
|
`go_back`: Navigate to the previously viewed page. |
|
`go_forward`: Navigate to the next page (if a previous 'go_back' action was performed). |
|
|
|
Completion Action: |
|
`stop [answer]`: Issue this action when you believe the task is complete. If the objective is to find a text-based answer, provide the answer in the bracket. If you believe the task is impossible to complete, provide the answer as "N/A" in the bracket. |
|
``` |
|
|
|
## Results on Benchmarks |
|
|
|
This model gets the following results on WebArena and WebVoyager: |
|
|
|
| Model | WebArena (SR) | WebVoyager (SR) | |
|
|------------------------|--------------:|---------------:| |
|
| **GPT-4** | **14.1** | **33.5** | |
|
| **llama8b-nnetnav-wa** | **16.3** | **28.1** | |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
### **Bias** |
|
As with all ML models, **Llama8b-NNetNav-WA** inherits biases from its training data. Since the dataset is collected via unsupervised exploration on self-hosted WebArena websites, it will reflect biases present in website structures, navigation flows, and content representations. |
|
|
|
- **Selection Bias:** The model is trained on Self-hosted websites that mimic reddit, github, google maps, simple e-commerce websites and CMS websites. This model is likely to struggle with websites with modern layouts seen on live websites. |
|
- **Demographic Bias:** WebArena self-hosted websites over-represent Western English-speaking users, and the model may perform worse on non-English or culturally distinct websites. |
|
- Example: A model trained mostly on U.S. e-commerce sites may navigate amazon.com effectively but may struggle with Flipkart (India) or Rakuten (Japan). |
|
|
|
If you are interested in training a NNetNav based agent for your own domain, please check out our [codebase](https://github.com/MurtyShikhar/NNetnav). Or if you're interested in a model that has been shown to work well on a variety of live websites, please check out [LLama8b-NNetNav-Live](https://huggingface.co/stanfordnlp/llama8b-nnetnav-live) |
|
|
|
### **Risks** |
|
#### **1. Unintended Actions** |
|
The model operates by executing web actions based on textual observation spaces, which may lead to unintended consequences when dealing with ambiguous or poorly structured websites. |
|
|
|
- If instructed to "delete all spam messages in my inbox," but the website has unusual button placement in the AXTree, the model might mistakenly delete important emails instead. |
|
- If asked to "buy the cheapest laptop on Amazon," the model might select an accessory instead of an actual laptop if the AXTree of the listing page has misleading layout |
|
|
|
#### **2. Security & Privacy Risks** |
|
Since the model interacts with external web content, there are significant risks related to unintentional data exposure, credential leaks, and interaction with harmful content. |
|
|
|
- If asked to "log into my Gmail and check unread emails," the model may type and submit credentials without realizing it, potentially exposing passwords. |
|
- A user asking the model to "search for free software downloads" might inadvertently lead to interactions with phishing or malware-hosting sites. |
|
|
|
#### **3. Adversarial Manipulation** |
|
Malicious websites can deceive the model by using **dark patterns**—UI/UX tricks that mislead users (or bots). |
|
|
|
- A fraudulent website may create **fake "Close" buttons** in the AXTree that actually trigger **downloads or pop-ups**. The model, thinking it's closing a window, may instead **click a malicious link**. |
|
- If asked to "unsubscribe from a newsletter," but the page uses **misleading button labels** in the AXTree (e.g., "Unsubscribe" actually means "Resubscribe"), the model could perform the opposite action. |
|
|
|
#### **4. Legal & Ethical Considerations** |
|
Web navigation often involves handling user-generated content, news, and e-commerce transactions, all of which pose ethical and legal challenges. |
|
|
|
- If instructed to "find the latest election results," the model might click on a misleading news source, potentially spreading misinformation. |
|
- If asked to "find the cheapest flight ticket," it could unintentionally violate terms of service by scraping restricted airline data. |
|
|
|
### **Limitations** |
|
#### **1. Generalization to Unseen Websites** |
|
This model is trained via interaction on 5 self-hosted WebArena websites, and is known to struggle on real, live websites. Please check out [LLama8b-NNetNav-Live](https://huggingface.co/stanfordnlp/llama8b-nnetnav-live) for a model that performs better on live websites. |
|
|
|
#### **2. Instruction Sensitivity** |
|
Vague instructions can lead to unintended actions. |
|
|
|
- "Find me the best laptop for gaming" is **subjective**, and the model might select a **random option** instead of following some criteria (e.g., GPU, refresh rate). |
|
|
|
#### **3. Performance on Long-Horizon Tasks** |
|
The model may struggle when tasks require **deep memory retention, complex multi-step planning, or backtracking**. |
|
|
|
- *Example:* When booking a hotel on a travel website, the model might navigate **through multiple filters and options** but forget previous selections when reaching the checkout page. |
|
|
|
#### **4. Token Limitations** |
|
The model's **maximum sequence length of 20k tokens** limits its ability to handle long, continuous web interactions. |
|
|
|
- *Example:* When filling a very long multi-step form, the model might forget earlier responses, leading to errors. |
|
|
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
TODO |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
This model was trained with SFT on the [NNetnav-WA](https://huggingface.co/datasets/stanfordnlp/nnetnav-wa) dataset, which is comprised of synthetic demonstrations entirely from self-hosted websites. |
|
|
|
### Training Procedure |
|
|
|
This model was trained for 2 epochs (roughly 4k gradient steps) with a batch size of 128, and a maximum sequence length of 20000. |
|
|
|
## Environmental Impact |
|
|
|
- **Hardware Type:** 4 H100 GPUs (80G) |
|
- **Hours used:** Roughly 2 days. |
|
- **Cloud Provider:** Stanford compute. |
|
- **Compute Region:** Stanford energy grid. |
|
|
|
## Technical Specifications |
|
|
|
### Hardware |
|
|
|
This model was trained on 4 H100s. |
|
|
|
### Software |
|
|
|
This model was fine-tuned with [Open-Instruct](https://github.com/allenai/open-instruct/tree/main) |
|
|
|
|
|
## Model Card Authors |
|
|
|
Shikhar Murty |
|
|
|
## Model Card Contact |
|
|
|
[email protected] |
|
|
