App Store Scraper
Comprehensive toolkit for retrieving structured data from Apple's App Store and iTunes APIs using curl and jq. All endpoints return JSON that can be parsed and formatted for analysis.
Quick Start
Tools Required:
-
curl for HTTP requests
-
jq for JSON parsing and formatting
Example:
Get app details
curl -s "https://itunes.apple.com/lookup?id=553834731&entity=software" |
jq '.results[0] | {name: .trackName, rating: .averageUserRating}'
Decision Trees
Use these decision trees to quickly find the right endpoint for your needs.
🎯 Decision Tree 1: What Data Do You Need?
What information do you need? │ ├─ App Metadata (name, price, developer, description) │ ├─ I know the app ID → App Lookup │ ├─ I know the bundle ID → App Lookup │ └─ I need to search by keyword → App Search │ ├─ App Discovery │ ├─ Browse top apps or categories → App Lists/Feeds │ ├─ Search by keyword → App Search │ ├─ Find all apps by a developer → Developer Apps │ ├─ Get search suggestions → Search Suggestions │ └─ Find similar apps → Similar Apps │ └─ User Feedback ├─ Read user reviews → Reviews └─ Get rating distribution (1-5 stars) → Ratings Histogram
🔍 Decision Tree 2: How Do You Identify the App?
How do you identify the app? │ ├─ I have the numeric app ID (e.g., 553834731) │ ├─ Get app details → App Lookup │ ├─ Get reviews → Reviews │ ├─ Get ratings breakdown → Ratings Histogram │ └─ Find similar apps → Similar Apps │ ├─ I have the bundle ID (e.g., com.apple.Numbers) │ └─ Get app details → App Lookup │ ├─ I have the developer ID │ └─ Get all apps by developer → Developer Apps │ ├─ I only know the app name │ └─ Search by keyword → App Search │ └─ I want to explore ├─ Browse by category → App Lists/Feeds └─ Get search suggestions → Search Suggestions
📊 Decision Tree 3: What Action Do You Want to Perform?
What do you want to do? │ ├─ Analyze a Specific App │ ├─ Get comprehensive metadata → App Lookup │ ├─ Read user feedback → Reviews │ ├─ Analyze rating distribution → Ratings Histogram │ └─ Find competitors/alternatives → Similar Apps │ ├─ Market Research │ ├─ Track top apps in categories → App Lists/Feeds │ ├─ Analyze developer portfolios → Developer Apps │ └─ Search by keyword/category → App Search │ └─ Monitor Changes ├─ Track rating changes → Ratings Histogram ├─ Monitor new reviews → Reviews └─ Watch top charts → App Lists/Feeds
🌍 Decision Tree 4: Regional & Multi-App Queries
Do you need region-specific or multi-app data? │ ├─ Multi-Region Analysis │ ├─ Compare ratings across regions → Ratings Histogram │ └─ Get reviews from different regions → Reviews │ ├─ Bulk Operations │ ├─ Lookup multiple apps → App Lookup │ └─ Search and filter results → App Search │ └─ Category Browsing └─ Browse by specific categories → App Lists/Feeds
📚 Documentation Index
Endpoints (API-Based)
Reliable, structured API endpoints that return JSON:
Endpoint Description File
App Lookup Get detailed app info by ID or bundle ID endpoints/app-lookup.md
App Search Search apps by keyword with filters endpoints/search.md
App Lists/Feeds Browse top apps and categories endpoints/lists.md
Developer Apps Get all apps by a specific developer endpoints/developer.md
Reviews Fetch paginated user reviews endpoints/reviews.md
Ratings Histogram Get 1-5 star rating breakdown endpoints/ratings.md
Search Suggestions Get autocomplete search hints endpoints/suggestions.md
Endpoints (Web Scraping)
Requires HTML parsing, less reliable:
Endpoint Description File
Similar Apps Find related apps (web scraping) endpoints/similar.md
Common Use Cases
Use Case 1: Competitive Analysis
1. Search for competitor apps
2. Get detailed info for top results
3. Compare ratings, features, pricing
4. Analyze user reviews
See: App Search → App Lookup → Reviews
Use Case 2: App Monitoring
1. Get current app state
2. Track rating changes over time
3. Monitor new reviews
4. Alert on rating drops
See: App Lookup → Ratings Histogram → Reviews
Use Case 3: Market Research
1. Browse top apps in category
2. Analyze pricing trends
3. Study feature patterns
4. Identify gaps in market
See: App Lists/Feeds → App Search
Use Case 4: Developer Portfolio Analysis
1. Find developer ID from an app
2. Get all apps by developer
3. Compare performance across portfolio
4. Track developer strategy
See: App Lookup → Developer Apps
Quick Reference
Essential Parameters
-
country - Market code (default: us )
-
entity - Always use software for apps
-
limit - Max results (varies by endpoint)
-
lang - Language preference (e.g., en-US , ja-JP )
Common Country Codes
Code Country Code Country
us
United States gb
United Kingdom
de
Germany fr
France
jp
Japan au
Australia
ca
Canada es
Spain
it
Italy br
Brazil
in
India mx
Mexico
kr
South Korea cn
China
Common Commands
Get app by ID
curl -s "https://itunes.apple.com/lookup?id=553834731&entity=software" | jq '.results[0]'
Search apps
curl -s "https://itunes.apple.com/search?term=weather&media=software&entity=software&limit=10"
Top free apps
curl -s "https://itunes.apple.com/us/rss/topfreeapplications/limit=25/json"
Recent reviews
curl -s "https://itunes.apple.com/us/rss/customerreviews/page=1/id=553834731/sortby=mostrecent/json"
Best Practices
-
Validate responses before processing with jq empty
-
Implement caching to reduce API load
-
Add rate limiting (1-2s between requests)
-
Batch requests when possible using comma-separated IDs
-
Handle errors gracefully with retries and fallbacks
API Limitations
-
Rate limiting: No official limits, but be respectful (1-2s between requests)
-
Pagination: Limited on some endpoints (max 200 results for search)
-
History: Only current version data available via API
-
Web scraping: Required for similar apps (unreliable, structure may change)
External Resources
-
iTunes Search API Documentation
-
RSS Feed Generator
-
jq Documentation
Start Here: Use the decision trees above to find the right endpoint for your needs, then follow the links to detailed documentation.