R18dev SQL Scraper

Plugins Other Projects
, ,
1
:placard: Summary This scraper requires access to a PostgreSQL server with R18.dev data.
:link: Repository https://github.com/javstash/R18dev_SQL

R18dev SQL Scraper

Database Setup

This scraper requires access to a PostgreSQL server with R18.dev data.

A docker compose file and a bash script to fetch and import the latest R18.dev database dump can be found under docker/. Simply run docker compose up --build -d and then docker exec -it postgres /setup_db.sh to set up a PostgreSQL database with R18.dev data. To update your database to the latest snapshot, rerun docker exec -it postgres /setup_db.sh after restarting the docker image.

If you already have PostgreSQL installed, you can find the latest R18.dev database dump here.

Initial setup

  1. Run pip install psycopg2-binary to install the psycopg2 library.
  2. Adjust the PostgreSQL connection string in R18dev_SQL.py if you are using your own PostgreSQL setup
  3. ???