fpassthru broken on OS X

Update: bug is now fixed. See [my new post][3]

I’ve gotten a few strange bug reports about broken downloads in SabreDAV over a certain size. I was admitedly a bit sceptical, because this always worked pretty well.

I’ve unable so far to confirm this on other systems, but I’m very curious what other people are seeing.

To test, create a large file. The following statement creates a 5GB file:

dd if=/dev/zero of=5gigs bs=1024 count=5242880

Next, we want to stream this with fpassthru(), the following script should do this:

<?php

$file = __DIR__ . '/5gigs';

$h = fopen($file, 'r');
fpassthru($h);

I’ve tried this at a few sizes:

  • At 1.5 GB the download succeeds
  • At 3 GB and 6GB the script crashes and I’m not getting an error message.
  • At 5GB I’m consistently only receiving 1 GB worth of data, before the script dies without warning.

This is tested on PHP 5.5.7 on OS X 10.9, and the behavior is the same through apache, or on the command line. If the previous file was called ‘foo.php’, you can easily test by running:

php foo.php > output

Output buffering and max execution time is off. Before I’m reporting a bug to bugs.php.net, I would love to know if I’m the only one out there seeing this.

So unless I’m making a mistake somewhere, else:

Do not use fpassthru on OS X if you want to support large files!

The following two workarounds always workthough:

<?php

$file = __DIR__ . '/5gigs';

$h = fopen($file, 'r');

while(!feof($h)) {
    echo fread($h, 4096);
}

And my personal preference:

<?php

$file = __DIR__ . '/5gigs';

$h = fopen($file, 'r');
file_put_contents('php://output', $h);

PHP Bug

Now confirmed on 5.5.9 and opened a php bug report: https://bugs.php.net/bug.php?id=66736

Web mentions

Comments

  • SM

    <p>fpassthru (eventually) uses size_t as file size if stream is mmap-able (which regular files usually are). size_it is 32-bit in 32-bit binaries, so if your PHP build is 32-bit, it's up to 4G. Or, more precisely, you get 5G number stuffed into 4G variable, which gives you ~1G. fread, etc. don't care too much about overall file size as they just try to read in small chunks. File offset is probably wrong somewhere but fread doesn't care.</p>
    • Evert

      Evert

      <p>My PHP_INT_SIZE = 8, so I'm definitely good in this regard. I don't think there's any reason these days to compile to 32-bit.</p>
  • BenBen

    <p>Did you try readfile($file)? Its my favorite way of tackling this sort of things and according to <a href="http://www.garfieldtech.com/blog/readfile-memory" rel="nofollow noopener" title="http://www.garfieldtech.com/blog/readfile-memory">http://www.garfieldtech.com...</a> it should be working and be performant</p>
    • Evert

      Evert

      <p>In my specific usecase, there is no 'filename', there's just a stream that can come from anywhere. See here:</p><p><a href="https://github.com/fruux/sabre-http/blob/master/lib/Sabre/HTTP/Sapi.php#L56" rel="nofollow noopener" title="https://github.com/fruux/sabre-http/blob/master/lib/Sabre/HTTP/Sapi.php#L56">https://github.com/fruux/sa...</a></p>
      • Guest

        <p>Aha, that makes sense :)</p>
  • Daniel Lowrey

    <p>If you're serving files of this size you had better be using the X-Sendfile header (or equivalent). Passing large files through userland like this is a serious performance WTF. No one should be doing this in the first place.</p>
    • Evert

      Evert

      <p>Streams can originate from everywhere, and I don't discriminate where. I agree that it would be a good bonus feature to also allow files to be served using X-Sendfile *if* the stream represents a file on a filesystem, but this is not the assumption.</p>
      • Daniel Lowrey

        <p>Ah, makes sense. I was initially horrified by the thought of someone streaming multi-Gb filesystem resources in chunks through userland :)</p>