phoenix
phoenixPhoenix - Your AI Assistant
Inactive session will be ended in 24 hours
End Session

Python Developer for UK Council Planning Portal Scraper

  • Posted:
  • Proposals: 31
  • Remote
  • #4515720
  • OPPORTUNITY
  • Awarded
Safeer H.Aurel H.Tosin O.Yassine A.M. C. + 26 others have already sent a proposal.
  • 5

Description

Experience Level: Expert
I run an architectural practice in the UK. We are looking for an experienced Python developer to build a robust internal tool (MVP) that automates our planning precedent research.

The goal is to scrape planning application data from two UK council portals (Hillingdon and Three Rivers). Both councils use variations of the standard Idox Public Access system.

The Required Workflow:

Navigate & Search: The script needs to bypass standard cookie banners and menus, navigate the planning portal, and search for a specific Application Reference or Street Name.

Data Extraction: Extract structured details (Reference, Address, Proposal Description, Decision Status, Decision Date).

Smart Document Handling (Crucial): Navigate to the "Documents" tab (often inside an iframe) and do the following:

Download: Physically download ONLY the "Officer's Report" and "Decision Notice" PDFs (bypassing the browser's PDF viewer).

Link Only: For all other documents (especially Architectural Drawings and Plans), do NOT download them. Instead, extract the direct URL/link to the document.

Database Integration: Push the structured data to our database (Airtable via API). Securely upload the downloaded Officer's Reports as file attachments, and paste the direct URLs for the drawings into a separate column.

Crucial Technical Challenges (Please read before bidding):
We have attempted a basic Playwright script but hit several anti-scraping and structural barriers. Your solution must cleanly handle:

Nested Iframes: The document lists are contained within an idox-iframe.

File Interception: Bypassing standard browser PDF rendering by pulling the raw file stream directly using session requests/cookies (to avoid Target closed exceptions).

Session Management & Politeness: Handling potential rate limits or dynamic ID changes. The scraper must be programmed to run politely (rate-limited) to respect council server loads.

Deliverables:

Clean, documented Python code (using Playwright, Scrapy, or Requests/BeautifulSoup).

API integration that pushes the structured data, uploads the specific PDFs, and saves the URLs to our Airtable base.

Simple instructions for running the script locally or setting it up on a basic schedule (cron job/cloud function).

Please answer the following in your bid:

Have you successfully scraped UK Council Idox systems or similar public portals before?

How will you prevent the browser's PDF viewer from intercepting the file download for the Officer's Reports?

What is your estimated timeline and cost for this MVP?

New Proposal

Login to your account and send a proposal now to get this project.

Log in

Clarification Board Ask a Question

  • VCONN P.

    Hi Jag,

    You’ve already identified the parts that usually make these portals difficult, particularly the iframe/document handling, so just a few points before we scope the MVP:

    1. Will you provide the Airtable base/schema, or should we structure the required tables and fields?

    2. Should the MVP process one application/search at a time, or a list of references/streets in one run?

    3. Roughly how many applications do you expect to process in a typical day?

    4. Does the tool need to work only with Hillingdon and Three Rivers for the MVP, or should we structure it so other Idox councils can be added later?

    For two portals + document identification/download + session handling + Airtable integration + testing, I would estimate around £500–£750 fixed and 7–10 working days.

    Looking forward to discussing further.

    Best Regards.

  • Asad C.

    -/ Should a street-name search return and process every matching planning application, or should the tool first present the matches so a user can select specific applications?

    -/ How should the scraper identify Officer Reports and Decision Notices when councils use different document titles such as “Delegated Report,” “Committee Report,” “Decision,” or “Decision Letter”? Do you already have a list of expected naming variations?

    -/ For drawings and plans, do you want every document URL stored individually, including title/type/date, or is one Airtable field containing a consolidated list of drawing links sufficient?

    -/ For downloaded PDFs, should the system store only the Airtable attachment, or do you also need a local/cloud backup storage location (such as AWS S3 or Google Drive)?

    -/ Are there any specific Airtable fields, views, automation rules, or existing workflows that this data needs to connect with?

  • CYTNHIA W.

    For the Airtable output, do you want each planning application treated as a persistent record that the scraper can revisit and update when the council later adds or changes documents/decision data, or is the MVP intended as a one-time extraction per application? I’m asking because I can design the scraper around idempotent upserts from the start, which would prevent duplicate Airtable records and make scheduled monitoring straightforward later.