Читаю серию из 5-ти статей "five life jackets..." - подробный пример Scrapy

Здесь много полезного. Начиная с пауков, советов типа $x(//div), подробно расписан процесс подключения Postgres... Скачал репозиторий. Посмотрел ссылки из статей, добавил "Scrapy Contracts Evolution"... Здесь только ссылки (штук 20)

five life jackets to throw to the new coder
github new-coder
Using Scrapy to extract data from Detail Pages
Scrapy, scrapping data inside a javascript
pyquery: a jquery-like library for python


What just happened under the hood?

Working with relative XPaths
Getting Started with Postgres Plus® Tutorial for Windows

Scrapy Unit Testing
Scrapy Contracts Evolution

Amazon Kindle Price Alerts
Web Scraping 101 with Python
Web scraping Amazon and Rotten Tomatoes
OpenShift Online Pricing

five life jackets to throw to the new coder¶

Introduction to Web Scraping using Scrapy and postgres
Part 0: Setup – Initial setup of your scraper environment.
Part 1: Scraper Setup – Building the scraper setup portion of the tutorial.
Part 2: Writing our Spider – Writing the spider portion of our scraper.
Part 3: Setting up SQLAlchemy – Setting up our data models using SQLAlchemy.
Part 4: Pipelining Data – Pipelining our scraped data into our Postgres database.
Part 5: Running our Scraper – Putting all the pieces together to scrape our data.
Extended – Scrapers that are used in real life, and where to go from here.

BOOKS, READINGS, AND WEBSITES¶

• Scraping Amazon and Rotten Tomatoes
• Web Scraping 101 with Python
• A package for scraping, storing, and sharing legislative information

Посты чуть ниже также могут вас заинтересовать