Web Scrapping - Data Collection

Data Scraping and Training for a GPT Chatbot



Introduction:

We are seeking an experienced freelancer to collect, structure, and utilize relevant data from various online sources to train a chatbot based on the OpenAI API. The chatbot will be integrated into our WordPress website, and must provide accurate and quick responses on topics related to aesthetic medicine and the treatments offered by our clinic.



Main Tasks:



1. Data Scraping

• Identify and list relevant websites containing public information about aesthetic medicine (FAQs, treatment descriptions, etc.).

• Collect data using appropriate scraping tools while adhering to regulations (legal notices, terms of use).

• Organize the collected data in a structured format such as JSON or CSV.



2. Data Structuring and Preparation for Training

• Filter and clean the data to ensure quality and relevance.

• Structure the data into question-answer pairs, descriptions, or specific contexts.

• Prepare a training file in JSONL format for use with the OpenAI API.



3. Documentation and Handover

• Provide detailed documentation on the scraping, data cleaning.

• Include recommendations for regular updates to the data and future model re-training.



Required Skills:

• Expertise in web scraping with tools such as Beautiful Soup, Scrapy, or Octoparse.

• Experience in data manipulation (cleaning, structuring, JSON/CSV formats).

• Strong understanding of machine learning concepts and natural language processing (NLP).

• Strict adherence to regulations and best practices for data collection and usage (GDPR, legal notices).



Deliverables:

• Collected, cleaned, and structured data in JSON or CSV format.

• A training file in JSONL format ready for use with the OpenAI API.

• Comprehensive documentation of the process, including tools and methodologies used.



Selection Process:

1. Review of proposals and portfolios.



To Apply:

If you are interested in this project and possess the required skills, please send us:

• Your CV or portfolio.

• A technical proposal describing your approach to scraping (and training ?).

• A budget estimate.