subscribe

Following redirects with Curl in PHP.

As a good web citizen, I try to always follow redirects. Not just in my browser, where I actually don’t have all that much control over things, but also a consumer of web services.

When doing requests with CURL, redirects are not followed by default.

<?php

$curl = curl_init('http://example.org/someredirect');
curl_setopt($curl, CURLOPT_POSTFIELDS, "foo");
curl_setopt($curl, CURLOPT_POST, true);

curl_exec($curl);

?>

Assuming the given url actually redirects like this:

HTTP/1.1 301 Moved Permanently
Location: /newendpoint

Curl will automatically just stop. To make it follow redirects, the FOLLOWLOCATION setting is needed, as such:

<?php

$curl = curl_init('http://example.org/someredirect');
curl_setopt($curl, CURLOPT_POSTFIELDS, "foo");
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_POST, true);

curl_exec($curl);

?>

CURLOPT_FOLLOWLOCATION will follow the redirects up to 5 times (by default).

However, if you look at the second request, it actually does a GET request after the POST.

GET /newendpoint HTTP/1.1

This is also the default behavior for browsers, but actually non-conforming with the HTTP standard, and also not desirable for consumers of web services.

To fix this, all you have to do is use CURLOPT_CUSTOMREQUEST instead of CURLOPT_POST:

<?php

$curl = curl_init('http://example.org/someredirect');
curl_setopt($curl, CURLOPT_POSTFIELDS, "foo");
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");

curl_exec($curl);

?>

Streams

After doing this, the secondary request will be a POST request as well. There’s one more issue though, if you were doing a POST or a PUT request you probably had a request body attached.

There’s two ways to supply a request body, as a string or as a stream. If we were uploading a file it makes much more sense to use a stream, because it unlike posting a string, a stream doesn’t have to be kept in memory.

To upload a stream with curl, you need CURLOPT_PUT and CURLOPT_INFILE. Don’t let the name CURLOPT_PUT fool you, it’s use for every request, and without CURLOPT_PUT, CURLOPT_INFILE is ignored.

For example, this is how we could upload a large file using POST.

<?php

$curl = curl_init('http://example.org/someredirect');
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_PUT, true);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($curl, CURLOPT_INFILE, fopen('largefile.json', 'r'));

curl_exec($curl);

?>

This will work great, unless the target location redirects. If it does, curl will throw the following error:

Necessary data rewind wasn't possible (code #65)

This seems to be related to PHP bug #47204.

Basically this means that you cannot use CURLOPT_INFILE and CURLOPT_FOLLOWLOCATION together. There’s two alternatives:

  1. Don’t use CURLOPT_INFILE, but send the request body as a string instead, with CURLOPT_POSTFIELDS.
  2. Don’t use CURLOPT_FOLLOWLOCATION, but instead manually check if the response was a 3xx redirect and manually follow each hop.

Strings

Using CURLOPT_POSTFIELDS you can supply a request body as a string. Lets try to upload our earlier failed request using that method:

<?php

$curl = curl_init('http://example.org/someredirect');
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($curl, CURLOPT_POSTFIELDS, file_get_contents('largefile.json'));

curl_exec($curl);

?>

This also will not work exactly as you expect. While the second request to /someredirect will still be a POST request, it will be sent with an empty request body.

To fix this, use the undocumented CURLOPT_POSTREDIR option.

<?php

$curl = curl_init('http://example.org/someredirect');
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($curl, CURLOPT_POSTFIELDS, file_get_contents('largefile.json'));
curl_setopt($curl, CURLOPT_POSTREDIR, 3);

curl_exec($curl);

?>

According to the PHP changelog, this was added in PHP 5.3.2, and according to PHP bug #49571 there are four possible values:

0 -> do not set any behavior
1 -> follow redirect with the same type of request only for 301 redirects.
2 -> follow redirect with the same type of request only for 302 redirects.
3 -> follow redirect with the same type of request both for 301 and 302 redirects.

Web mentions

Comments

  • MeadSteve •

    I guess when you get a 301/308 it'd normally be worth logging something as well, for the maintainer of the code to take some action. Versus a 302/307 where you'd be happy for the code to do this silently.

  • Ed Dikotope •

    Best answer to my problem over the entire internet. Thank you very much.

  • Tom Binga •

    You helped solved a problem that's been holding me back for a week or so now. Thank you!

  • Marcos Saturno •

    Very good info!

    I'm still having problems when trying to write onto web HDFS (hadoop) from a PHP script. If I make a PUT request passing CREATE as a parameter, it will provide me a 307 temporary redirect, so i'd need to make a second PUT request to the new URL provided. I'm not able to follow it or at least slipt the redirect URL from the HTTP Response. Could you please help me with it?

    DOC:
    http://hadoop.apache.org/do...

    I'm using something like:

    $options = array(

    // CURLOPT_PUT => true,

    CURLOPT_HEADER => true,

    CURLOPT_CUSTOMREQUEST => "PUT",

    CURLOPT_FOLLOWLOCATION => true

    );

    $ch = curl_init('<host>:<port>/webhdfs/v1/user/USER?op=CREATE');

    // Execute
    curl_setopt_array($ch, $options);

    curl_exec($ch);

    //echo curl_errno($ch);

    if(!curl_errno($ch))
    {

    $info = curl_getinfo($ch);

    echo 'Took ' . $info['total_time'] . ' seconds to send a request to ' . $info['url'];

    }

    // Close handle

    curl_close($ch);

    ?>

    • Marcos Saturno •

      *I commented the //CURLOPT_PUT, but also tested with both on. ;-)

      • Evert

        Evert •

        Does your request not have a body? I wonder if that messes things up. You may want to PUT an empty string instead of nothing at all, because curl may fall back to 'GET' behavior (although with a PUT method).

        • Marcos Saturno •

          Hmm, good tip!

          But actually I found some classes ready to use WEB HDFS:
          https://github.com/simpleen...

          Tx for the help, anyways!

          Regards,

          Marcos.

  • Roger Qiu •

    I don't feel secure knowing my request might be redirected to some place I don't know.

  • Shanly •

    Awesome post! This solved my problem, thanks.

  • Bub •

    Well done!

  • Arlanthir •

    Thank you for this.

  • Ajai •

    i am doing an request using Advanced REST Client and i find some redirects (not cached) , but i get the actual response. But when i try the same using PHP curl it doesnt work , i dont get the output , but the redirected output is displayed.

    Please can any one help me

  • richyrich •

    Thank you so much for this article, saved my time. All the best!

  • Hassan Nomani Alvi •

    I am trying to post data using curl.After posting I would like to go to the url.In other words I am trying to get same functionality as we get with form method="post" and action="someurl.php" .How to do this?Thanks in advance.