> ## Documentation Index
> Fetch the complete documentation index at: https://docs.utopikai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Connector

> Access knowledge from Web Pages

## How it works

The Web Connector scrapes sites based on a base URL.

* It only indexes files from the same domain and containing the same base path.
* It will index pages reachable via hyperlinks from the base URL.
* The text contents are cleaned up via some heuristics and some metadata such as the page Title is extracted.

## Setting up

### Authorization

* As long as the page is reachable, no additional authorization is necessary.

### Indexing

1. Navigate to the Admin Dashboard and select the **Web** Connector.
2. Input the base URL to index and click on Index.

<img src="https://mintcdn.com/utopikai/5TAKgT3IAyEwTGK_/images/connectors/web/WebConnector.png?fit=max&auto=format&n=5TAKgT3IAyEwTGK_&q=85&s=8a1f5e2d09ce2864d78ede16a6df2f98" alt="WebConnector" width="1895" height="1254" data-path="images/connectors/web/WebConnector.png" />

To see the status of the indexing, visit the Connectors Status page (top left).

<img src="https://mintcdn.com/utopikai/5TAKgT3IAyEwTGK_/images/connectors/web/WebConnectorStatus.png?fit=max&auto=format&n=5TAKgT3IAyEwTGK_&q=85&s=2538503c97a36af0361f1ef026be20ef" alt="WebConnectorStatus" width="2491" height="989" data-path="images/connectors/web/WebConnectorStatus.png" />