Python Developer for UK Council Planning Portal Scraper

I run an architectural practice in the UK. We are looking for an experienced Python developer to build a robust internal tool (MVP) that automates our planning precedent research.

The goal is to scrape planning application data from two UK council portals (Hillingdon and Three Rivers). Both councils use variations of the standard Idox Public Access system.

The Required Workflow:

Navigate & Search: The script needs to bypass standard cookie banners and menus, navigate the planning portal, and search for a specific Application Reference or Street Name.

Data Extraction: Extract structured details (Reference, Address, Proposal Description, Decision Status, Decision Date).

Smart Document Handling (Crucial): Navigate to the "Documents" tab (often inside an iframe) and do the following:

Download: Physically download ONLY the "Officer's Report" and "Decision Notice" PDFs (bypassing the browser's PDF viewer).

Link Only: For all other documents (especially Architectural Drawings and Plans), do NOT download them. Instead, extract the direct URL/link to the document.

Database Integration: Push the structured data to our database (Airtable via API). Securely upload the downloaded Officer's Reports as file attachments, and paste the direct URLs for the drawings into a separate column.

Crucial Technical Challenges (Please read before bidding):
We have attempted a basic Playwright script but hit several anti-scraping and structural barriers. Your solution must cleanly handle:

Nested Iframes: The document lists are contained within an idox-iframe.

File Interception: Bypassing standard browser PDF rendering by pulling the raw file stream directly using session requests/cookies (to avoid Target closed exceptions).

Session Management & Politeness: Handling potential rate limits or dynamic ID changes. The scraper must be programmed to run politely (rate-limited) to respect council server loads.

Deliverables:

Clean, documented Python code (using Playwright, Scrapy, or Requests/BeautifulSoup).

API integration that pushes the structured data, uploads the specific PDFs, and saves the URLs to our Airtable base.

Simple instructions for running the script locally or setting it up on a basic schedule (cron job/cloud function).

Please answer the following in your bid:

Have you successfully scraped UK Council Idox systems or similar public portals before?

How will you prevent the browser's PDF viewer from intercepting the file download for the Officer's Reports?

What is your estimated timeline and cost for this MVP?