Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Scraping httpstat.us/200 reliably triggers 500 responses from the Firecrawl API (FIR-519) #1029

Open
karolinepauls opened this issue Dec 31, 2024 · 2 comments
Open

[Bug] Scraping httpstat.us/200 reliably triggers 500 responses from the Firecrawl API (FIR-519) #1029

karolinepauls opened this issue Dec 31, 2024 · 2 comments
Assignees
@mogery
Labels
bug Something isn't working

Comments

@karolinepauls
Copy link

karolinepauls commented Dec 31, 2024 •
edited
Loading

Describe the Bug
Looks like empty scraped page responses cause 500 errors.

To Reproduce
Steps to reproduce the issue:
https://www.firecrawl.dev/playground?url=http%3A%2F%2Fhttpstat.us%2F200&mode=scrape

markdown:

curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
    -d '{
        "url": "http://httpstat.us/200",
                "formats": [ "markdown" ]
        }'

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>

rawHtml:

curl -X POST https://api.firecrawl.dev/v1/scrape     -H 'Content-Type: application/json'     -H "Authorization: Bearer $FIRECRAWL_API_KEY"     -d '{
        "url": "http://httpstat.us/200",
                "formats": [ "rawHtml" ]
        }'
{"success":false,"error":"(Internal server error) - All scraping engines failed! -- Double check the URL to make sure it's not broken. If the issue persists, contact us at [email protected]."}

Expected Behavior
OK API response containing empty content

Screenshots
image
@karolinepauls karolinepauls added the bug Something isn't working label Dec 31, 2024
@nickscamara
Copy link
Member

@mogery thoughts on this?

@mogery
Copy link
Member

mogery commented Jan 4, 2025

This is likely because the returned content is not long enough. We should probably be returning the short content anyways if the content is confirmed by multiple scraping engines.

@mogery mogery changed the title [Bug] Scraping httpstat.us/200 reliably triggers 500 responses from the Firecrawl API [Bug] Scraping httpstat.us/200 reliably triggers 500 responses from the Firecrawl API (FIR-519) Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mogery mogery

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nickscamara @karolinepauls @mogery