subscribe

fpassthru broken on OS X

Update: bug is now fixed. See [my new post][3]

I’ve gotten a few strange bug reports about broken downloads in SabreDAV over a certain size. I was admitedly a bit sceptical, because this always worked pretty well.

I’ve unable so far to confirm this on other systems, but I’m very curious what other people are seeing.

To test, create a large file. The following statement creates a 5GB file:

dd if=/dev/zero of=5gigs bs=1024 count=5242880

Next, we want to stream this with fpassthru(), the following script should do this:

<?php

$file = __DIR__ . '/5gigs';

$h = fopen($file, 'r');
fpassthru($h);

I’ve tried this at a few sizes:

  • At 1.5 GB the download succeeds
  • At 3 GB and 6GB the script crashes and I’m not getting an error message.
  • At 5GB I’m consistently only receiving 1 GB worth of data, before the script dies without warning.

This is tested on PHP 5.5.7 on OS X 10.9, and the behavior is the same through apache, or on the command line. If the previous file was called ‘foo.php’, you can easily test by running:

php foo.php > output

Output buffering and max execution time is off. Before I’m reporting a bug to bugs.php.net, I would love to know if I’m the only one out there seeing this.

So unless I’m making a mistake somewhere, else:

Do not use fpassthru on OS X if you want to support large files!

The following two workarounds always workthough:

<?php

$file = __DIR__ . '/5gigs';

$h = fopen($file, 'r');

while(!feof($h)) {
    echo fread($h, 4096);
}

And my personal preference:

<?php

$file = __DIR__ . '/5gigs';

$h = fopen($file, 'r');
file_put_contents('php://output', $h);

PHP Bug

Now confirmed on 5.5.9 and opened a php bug report: https://bugs.php.net/bug.php?id=66736

Web mentions

Comments

  • SM

    fpassthru (eventually) uses size_t as file size if stream is mmap-able (which regular files usually are). size_it is 32-bit in 32-bit binaries, so if your PHP build is 32-bit, it's up to 4G. Or, more precisely, you get 5G number stuffed into 4G variable, which gives you ~1G. fread, etc. don't care too much about overall file size as they just try to read in small chunks. File offset is probably wrong somewhere but fread doesn't care.

    • Evert

      Evert

      My PHP_INT_SIZE = 8, so I'm definitely good in this regard. I don't think there's any reason these days to compile to 32-bit.

  • BenBen

    Did you try readfile($file)? Its my favorite way of tackling this sort of things and according to http://www.garfieldtech.com... it should be working and be performant

    • Evert

      Evert

      In my specific usecase, there is no 'filename', there's just a stream that can come from anywhere. See here:

      https://github.com/fruux/sa...

      • Guest

        Aha, that makes sense :)

  • Daniel Lowrey

    If you're serving files of this size you had better be using the X-Sendfile header (or equivalent). Passing large files through userland like this is a serious performance WTF. No one should be doing this in the first place.

    • Evert

      Evert

      Streams can originate from everywhere, and I don't discriminate where. I agree that it would be a good bonus feature to also allow files to be served using X-Sendfile *if* the stream represents a file on a filesystem, but this is not the assumption.

      • Daniel Lowrey

        Ah, makes sense. I was initially horrified by the thought of someone streaming multi-Gb filesystem resources in chunks through userland :)