Data Scraper API

The Data Scraper API is a web service built with NestJS, TypeORM, and Puppeteer. It provides the following key functionalities:

• Automated Web Scraping: Extracts contact information (phone numbers, emails, addresses) from web pages.
• Search Service: Uses the Bing Search API to dynamically discover relevant URLs based on a search query.
• Database Storage: Saves scraped contact details in a PostgreSQL database.
• Telemetry Integration: Pushes scraping and search operation metrics to Observe Inc. for monitoring.
• Dockerized Deployment: Runs in a self-contained environment using Docker Compose.

Ensure you have the following installed on your system:

• Node.js (v22+)
• pnpm (v8+)
• Docker & Docker Compose
• PostgreSQL (optional, if running locally instead of Docker)

Create a .env file in the root directory with the following values:

# Database Configuration
DB_HOST=postgres
DB_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD=local
POSTGRES_DB=local

# Bing Search API
BING_API_KEY=your_bing_api_key
BING_SEARCH_URL=https://api.bing.microsoft.com/v7.0/search

# Google Search API
GOOGLE_API_KEY=google_api_key
GOOGLE_SEARCH_ENGINE_ID=google_search_engine_id
GOOGLE_SEARCH_URL=https://www.googleapis.com/customsearch/v1

# Observe Inc. Telemetry
OBSERVE_CUSTOMER=your_observe_customer_id
OBSERVE_TOKEN=your_observe_ingest_token
OBSERVE_DATASTREAM=your_datastream_name

docker compose up --build

This will:

• Build and start the NestJS API
• Start PostgreSQL and initialize the database with sample data
• Run all services inside Docker

To confirm that the database and sample data were created:

docker exec -it local psql -U user -d contactdb -c "SELECT * FROM \"contact-info\";"

This should return the pre-seeded sample contact information.

• Uses the Bing Search API to fetch URLs based on a search query.
• Returns a list of relevant websites for scraping.
• Pushes search metrics to Observe Inc..

• Uses Puppeteer to visit websites and extract contact details.
• Stores extracted data in the PostgreSQL database.
• Pushes scraping success/failure metrics to Observe Inc..

• Sends metrics (search success, scrape success, durations) to Observe Inc..
• Helps monitor system performance and failures.

• Manages PostgreSQL connection using TypeORM.
• Seeds initial sample data from seed.sql.
• Provides a repository for saving contact data.

POST /search - Searches the internet and scrapes the found URLs.

{
  "query": "senior living accommodations contact websites"
}

POST /scrape - Scrapes a list of URLs directly.

{
  "urls": ["https://example.com"]
}

pnpm run db:create    # Creates database schema
pnpm run db:drop      # Drops database schema
pnpm run db:seed      # Seeds sample data
pnpm run db:reseed    # Drops & re-creates database, then seeds
pnpm run db:migrate   # Runs pending migrations
pnpm run db:rollback  # Reverts last migration
pnpm run db:generate-migration  # Generates a new migration
pnpm run db:run-migrations      # Runs all pending migrations

pnpm run test         # Run unit tests
pnpm run test:e2e     # Run end-to-end tests

This Data Scraper API automates the process of discovering and extracting contact details from the internet. With Bing Search API, Puppeteer web scraping, and Observe Inc. telemetry, it ensures a scalable and well-monitored data extraction pipeline.

For any issues, refer to the logs, and check Observe Inc. dashboard for real-time monitoring!