How to Scrape Instagram Data with Python (Step-by-Step Tutorial)

To scrape Instagram data with Python, fetch its internal JSON responses instead of parsing HTML. This scraping approach covers only public profile and post fields, but endpoints change.

1. Find the GraphQL call in Chrome DevTools.
2. Replay the request in Python code.
3. Extract profile and post fields safely.
4. Save the results.
5. Avoid blocks while scraping with randomized pacing.

The tutorial uses requests and BeautifulSoup only where HTML fragments need parsing. Suppose an analyst collects follower counts and the latest 12 posts for 50 public Instagram brand accounts into a CSV by tomorrow.

What Instagram Data Can You Scrape?

Public Profile Data

From a public Instagram profile, you can extract basic identity and audience signals without logging in:

• username and full name
• bio text
• follower and following counts
• external links
• verification flag
• profile picture URL
• post count

These fields are visible to logged-out visitors.

Posts, Hashtags and Comments

• The post’s caption and publishing timestamp.
• Media URLs for each visible image or video.
• Hashtag text embedded in the caption.
• Like counts and comment counts.
• Top-level comments on public posts.
• These fields support engagement metrics and comments tracking.

What You Can't Scrape

An unauthenticated scraper sees only what a logged-out visitor sees. On Instagram, these remain unavailable:

• Private accounts.
• Direct messages.
• Complete follower lists.
• Most remaining data behind login.

Is It Legal to Scrape Instagram? Ethics and the Official API

Not legal advice. Meta restricts scraping public data. Minimize personal data. Prefer Graph API or official API.

Terms of Service and Public Data

Instagram’s terms of service and robots rules limit scraping. Public availability does not remove privacy obligations, including GDPR for personal data. For responsible scraping:

• Collect only needed public fields.
• Keep query volume very low and rate-limited.
• Bind each collection task to one defined purpose.
• Never republish or sell personal data.

Official Graph API vs Scraping

Use the official API when

Scraping fills gaps when

Set Up Your Python Environment

Install requests and BeautifulSoup

Install both libraries inside a clean Python virtual environment. Use pip to install them. requests handles the API calls. BeautifulSoup stays available only when an endpoint returns HTML instead of structured data. That keeps dependencies isolated and avoids global conflicts for later runs.

python -m venv ig-scraper# macOS/Linuxsource ig-scraper/bin/activate# Windows: ig-scraper\Scripts\activatepip install requests beautifulsoup4

import requestsfrom bs4 import BeautifulSoup

Create a Reusable Session with Browser-Like Headers

Create a reusable requests.Session so every call carries the same browser-like headers. Add a normal User-Agent, Accept-Language, and x-ig-app-id. Missing header values often produce empty responses during scraping. Keep these values consistent across requests to avoid obvious mismatches.

import requestssession = requests.Session()session.headers.update({    "User-Agent": "Mozilla/5.0",    "Accept-Language": "en-US,en",    "x-ig-app-id": "COPY_FROM_DEVTOOLS",    "Referer": "https://www.instagram.com/",})# Reuse this session for later callssession.trust_env = True

Find Instagram's Hidden API with Browser DevTools

Open DevTools (F12) and the Network Tab

1. Open a public profile page in Chrome while logged out to match an anonymous visitor.
2. Press F12 or right-click and choose Inspect to open DevTools.
3. Select the network tab and enable Preserve log before reloading.
4. Reload and keep the panel open while requests appear.

Spot the GraphQL and XHR Requests

In DevTools, filter Network by Fetch/XHR. Watch the API endpoints that return structured JSON.

• URL: contains web_profile_info or /graphql/query.
• Method: GET for profile data, POST for GraphQL.
• Payload: GraphQL variables identifying the profile or post.
• Response: Preview tab shows fields like follower counts, media, captions.

Copy the Request as cURL and Convert It to Python

• Right click the captured call and choose Copy as cURL.
• Keep the URL, method, headers, and query variables.
• Strip personal cookies, session tokens, and account identifiers.
• Rewrite the command as Python requests code and never commit secrets.

How Instagram's GraphQL API Works

Required Headers

Instagram rejects scraping calls that lack browser headers. Add these to every requests call:

• User-Agent: identifies a browser.
• x-ig-app-id: marks traffic as Instagram web.
• Accept-Language: matches page locale.
• Referer: shows the originating page.
• X-Requested-With: signals an XHR query.

Query Identifiers and Variables

Instagram queries use identifiers, not stable URLs.

• doc_id names a saved GraphQL query.
• variables carry the target shortcode, username, or cursor.
• Pagination repeats the query with the newest cursor.
• When GraphQL scraping fails, capture fresh identifiers in DevTools.

Scrape Instagram Profile Data with Python

Request the Profile Endpoint

Use the same requests session to call Instagram’s web_profile_info endpoint. Pass the target username as a query parameter. A successful reply returns a public profile object. If the status is 401, 403, or 429, stop and handle it before parsing. Return None or raise a clear error.

def get_profile(username, session):    url = "https://www.instagram.com/api/v1/users/web_profile_info/"    params = {"username": username}    try:        response = session.get(url, params=params, timeout=15)    except OSError as exc:        return {"error": str(exc), "username": username}    if response.status_code != 200:        return {"error": f"HTTP {response.status_code}", "username": username}    try:        payload = response.json()    except ValueError:        return {"error": "Invalid JSON", "username": username}    return payload.get("data", {}).get("user")

Extract Fields from the JSON Response

The response is a nested dictionary. Start at data, then the user object, and extract only the profile fields you need. Use .get() so missing values do not crash the parser. This returns one clean JSON record. Map full_name, biography, followers, following, posts, and is_verified into simple keys.

def parse_profile(payload):    user = (        payload        .get("data", {})        .get("user", {})    )    def count(key):        return (            user            .get(key, {})            .get("count")        )    return {        "username": user.get("username"),        "full_name": user.get("full_name"),        "biography": user.get("biography"),        "followers": count("edge_followed_by"),        "following": count("edge_follow"),        "posts": count("edge_owner_to_timeline_media"),        "verified": user.get("is_verified"),        "external_url": user.get("external_url"),    }

Handle Missing and Changed Fields

Instagram fields move or disappear without notice, so parse defensively and keep enough context to diagnose data issues.

• Use default values for absent keys.
• Log unexpected structures instead of crashing.
• Save raw responses for debugging.
• Mark missing profile fields for later review.

Scrape Instagram Posts and Paginate Through Them

Get the Latest Posts from a Profile

Read the latest timeline media under edge_owner_to_timeline_media in the profile response and extract the node fields below. Each edge usually represents one recent post, so keep identifiers for later joins. Store them in a flat row for analysis.

• shortcode
• caption text
• timestamp
• like and comment counts
• media URLs

Paginate with end_cursor

Instagram returns timeline pages through the same GraphQL query. Use this pagination loop while has_next_page is true. Pass end_cursor in variables. Sleep randomly between requests so rate limits are less likely. Stop at a fixed page cap. Save each batch. Re-check the cursor if rate limits appear.

import randomimport timedef scrape_posts(session, doc_id, username, max_posts=50):    url = "https://www.instagram.com/graphql/query/"    posts = []    cursor = None    while len(posts) < max_posts:        variables = {            "username": username,            "first": max_posts,            "after": cursor,        }        response = session.post(            url,            data={                "doc_id": doc_id,                "variables": variables,            },        )        response.raise_for_status()        timeline = response.json()["data"]["user"]["edge_owner_to_timeline_media"]        for edge in timeline.get("edges", []):            node = edge["node"]            caption_edges = node.get("edge_media_to_caption", {}).get("edges", [])            caption = caption_edges[0]["node"].get("text") if caption_edges else ""            posts.append({                "shortcode": node.get("shortcode"),                "caption": caption,                "taken_at": node.get("taken_at_timestamp"),                "likes": node.get("edge_liked_by", {}).get("count"),                "comments": node.get("edge_media_to_comment", {}).get("count"),                "display_url": node.get("display_url"),                "hashtags": [tag for tag in caption.split() if tag.startswith("#")],            })        page_info = timeline.get("page_info", {})        if not page_info.get("has_next_page"):            break        cursor = page_info.get("end_cursor")        time.sleep(random.random())    total = len(posts)    average_likes = sum(post["likes"] or 0 for post in posts) / total if total else None    average_comments = sum(post["comments"] or 0 for post in posts) / total if total else None    return posts, average_likes, average_comments

Scrape Instagram Comments

Comments sit behind the same GraphQL queries as posts. Use the post shortcode or media ID in the query variables, then paginate with the returned cursor.

For example, you can track comment volume and sentiment on a product-launch post without storing usernames. These engagement metrics measure reception. This keeps engagement metrics comparable across posts. Comment data is personal data, so scraping should minimize what it keeps.

import jsondef fetch_comment_page(session, doc_id, shortcode, cursor=None, page_size=None):    url = "https://www.instagram.com/graphql/query/"    variables = {"shortcode": shortcode}    if page_size:        variables["first"] = page_size    if cursor:        variables["after"] = cursor    payload = {        "doc_id": doc_id,        "variables": json.dumps(variables),    }    resp = session.post(url, data=payload)    resp.raise_for_status()    return resp.json()

Extract only stable fields:

• Text length
• Timestamp
• Like count
• Anonymized author ID

Save the raw cursor path for debugging and repeat requests slowly.

Save Your Scraped Data

Export to CSV

Write the scraped data to CSV with DictWriter. Add a scraped_at timestamp to every row, so repeated runs can track follower growth and post engagement over time for each account.

import csvfrom datetime import datetime, timezonedef save_csv(rows, filename):    if not rows:        return    rows = [{**row, "scraped_at": datetime.now(timezone.utc).isoformat()} for row in rows]    fieldnames = list(rows[0].keys())    with open(filename, "w", newline="", encoding="utf-8") as f:        writer = csv.DictWriter(f, fieldnames=fieldnames)        writer.writeheader()        writer.writerows(rows)

Store Results in a Database

Store scraped data in PostgreSQL or SQLite for repeated runs.

• Make username or shortcode a unique key.
• Upsert rows to replace stale records.
• Add timestamps to track each collection run.

Avoid Blocks: Rate Limits, Fingerprinting and Proxy Rotation

Rushing one IP triggers rate limits, browser fingerprinting, login redirects, failed scraping, and costly proxy rotation after bans.

Request Pattern Detection and Rate Limits

Instagram flags scraping that looks mechanical. Fixed delays, bursts, and high volume from one IP trigger rate limits. Repeated rate limits can block the IP and stop later scraping.

• Add random jitter between requests.
• Back off exponentially after a 429.
• Keep batches small.
• Cache responses already fetched.
• Pause when rate limits repeat.

Browser Fingerprinting and Header Consistency

Browser fingerprinting checks whether requests come from one coherent client. A mismatch between headers, TLS behavior, and cookies makes scraping easier to flag.

• Keep one User-Agent per session.
• Do not mix mobile and desktop headers.
• Reuse cookies for the same identity.
• Match language, platform, and referer values.

Rotate Proxies in Python

Pass an HTTP proxy dict to requests and rotate proxies between batches, backing off on 429. SOCKS5 support needs requests[socks] and socks5h://. This proxy rotation keeps batches on different IPs.

proxy_pool = [    {"http": "http://user:pass@host1:port", "https": "http://user:pass@host1:port"},    {"http": "socks5h://user:pass@host2:port", "https": "socks5h://user:pass@host2:port"},]for proxy in proxy_pool:    for attempt in range(4):        response = session.get(url, proxies=proxy, timeout=20)        if response.status_code == 429:            time.sleep((2 ** attempt) + random.uniform(0, 1))            continue        response.raise_for_status()        save(response.json())        break

Troubleshooting Common Scraper Errors

Login Redirects, 401 and 403

• Login page, 401, or 403 instead of expected data.
• Missing headers, a login-only resource, or a flagged IP.
• Recapture headers in DevTools, slow requests, or check whether your IP is flagged.

429 Too Many Requests

• 429 means Instagram received too many requests from this client.
• Pause with exponential backoff before retrying the same call.
• Lower concurrency and spread load across IPs to respect rate limits.

Empty or Changed Responses

• Symptom: Requests return empty payloads, blank arrays, or missing keys.
• Cause: Instagram changed the doc_id or response structure during a deployment.
• Fix: Re-capture the request in DevTools, then update parsing logic.

Build Your Own Scraper or Use a Scraping API?

The core trade-off: building gives control and cheap requests, but you own maintenance when web anti-bot rules or API endpoints change. A managed scraping API shifts that burden, at the cost of dependency and ongoing fees.

Build your own if…

• It is a weekend research project.
• You need exact fields only.
• You can patch Scrapy breakage quickly.

Use an API if…

• You run daily production jobs.
• You target 10,000 web profiles.
• Uptime matters more than control.

FAQ: Instagram Scraping with Python

Pace Python Instagram scraping with proxies and respect rate limits.

Is it possible to scrape Instagram data with Python?

Yes. Python can scrape public Instagram data by calling the same web endpoints the browser uses. Without login, access stays limited and usually requires careful request pacing.

Is there ready-made Python code for Instagram?

Yes, open-source Python libraries and snippets exist for Instagram scraping. However, they often break when endpoints or identifiers change. Knowing the DevTools method lets you update the requests quickly.

Can I scrape Instagram data without getting detected?

No technique guarantees stealth. Reduce risk with slow, randomized requests, consistent headers, small data volumes, and traffic spread across different IPs, including rotating proxies, to stay under rate limits.

Do I need to log in to scrape Instagram?

Public profile data is often reachable without a login for scraping. Follower lists, many comments, and private content usually require full authentication. Logging in adds real account-ban and terms-of-service risk.

Is AI or automated scraping illegal?

Legality depends on jurisdiction, whether data is personal, and the terms of service you accepted. GDPR may apply. For commercial use, consult a lawyer before deploying scrapers.

Key Takeaways

• Capture the actual Instagram request in DevTools before writing any Python code.
• Replay it with matching browser headers, minus personal cookies or secrets.
• Parse responses defensively, use safe lookups, sensible defaults, and log anomalies.
• Pace scraping, respect legal limits, and export each run to a timestamped CSV file for tracking.