Thursday, December 24, 2009

Progressive Upload with PHP and APC

A year or two ago, I was given the task to make a Progress Bar work with PHP. From my own research a few years ago, this task was impossible and so I thought I would need some sort of Flash + Javascript + PHP combination to make this happen.

Until now.

I found a nice recipe from IBM that actually makes this process much simpler. It's by using a simple PECL extension called Alternative PHP Cache (APC for short). This brilliantly simple extension allows us to FINALLY be able to monitor file upload progress! I read the tutorial and right away, I was throwing my fists into the air: it's finally possible to do progress bars with PHP WITHOUT THE NEED OF CGI !!!!

For anyone who has ever run into this issue, you are probably the only ones to truly respect this awesome new toy. What's even more bizarre is that this functionality has been out since PHP 5.2 and this is truly the first I've ever heard about it.

My problems though soon caught up with me. Apparently IBM's little "tutorial" is a little short-winded and doesn't exactly cover all the bases that are needed to make this new extension work. So, being the great guy I am, I decided to write a step-by-step tutorial on how to use this for the weak at heart.

==========================================================

*** INSTALLATION ***

First, you need to get the APC module. There are 2 downloads available:
Linux and Windows

* Please note that the Windows link above may be broken as the team is trying to move this box over to a new server... *

Next, you need to Install the modue.

-- FreeBSD/Linux Users only --
First things first: you'll need root to make all this work, so su into the root user.

After downloading the APC package (I downloaded the 3.0.18 stable bundle), you need to unpackage it. Once you have untar'ed the file, you need to cd into the apc directory and run the following set of commands:

phpize
./configure --enable-apc --enable-mmap
make install
cp modules/apc.so /usr/local/lib/php/
The phpize command is used to prepare the build environment for a PHP extension. If you don't have phpize installed, you will want to do so via RPM, ports, or whatever method your nix distro supports.

Here are some commands for a few linux distros:

Fedora Core x:
yum -y install php-devel
Gentoo Click here for the gentoo quick guide

FreeBSD Installed by default with PHP

I can't cover them all, but these 3 seem to be the most popular (along with Ubuntu but I'm not familiar with it)

Next, you'll need to open the php.ini file to make a few quick changes.

vi /usr/local/etc/php.ini
Locate where the Windows Extensions are loaded (yes I know you aren't using Winblows, but you'll see what I'm getting at next).

Below the ;extension=php_zip.dll line, add a new line and enter the following extension:

extension=apc.so
Return a few lines down and enter the following:
;;;;;;;;;;;;;;;;;;;;;;;
; APC LAYER CONTROL ;
;;;;;;;;;;;;;;;;;;;;;;;
apc.rfc1867=on
apc.max_file_size=200M
upload_max_filesize=200M
post_max_size=200M

Now, you do NOT have to use 200M. I simply did to test out uploading huge 191M video files to see if this progress bar truly works. Adjust this to suit your own personal needs.

Finally, save the file and exit by typing:

:wq!
You'll need the bang(!) because php.ini is by default a read-only file.

The last step now is just to restart apache:

apachectl restart
apachectl is the command that restarts Apache in FreeBSD. However, it may or may not be available in your distro/PATH (user's profile) so you may need to browse into that directory and run the appropriate server command used in your distro (example: /etc/rc.id/init.d/httpd -k restart).

Since I have never done this on windows, I'm just going to copy/paste IBM's installation method here. NOTICE: It's asking you to use the WAMP install package, but most developers here manually have their's installed. If you have issues, just post it here and we'll see if a solution can't be found.

--- WINDOWS USERS ONLY ---

APC is not enabled by default in PHP V5.2. Since the new hooks are a part of APC, we need to make sure to install the extension and make it available to the PHP interpreter. This is accomplished by downloading the php_apc extension files. In our case, we are using an installation of WAMP, a freely available packaged PHP for Windows®, which includes Apache and MySQL. It offers a nice user interface and is easy to manage with menus that support configuration options.

To set up APC on WAMP:

1. See Resources to download the libraries and WAMP.
2. Install WAMP.
3. Put the php_apc.dll file in the extensions folder for PHP. This is /php/ext by default.
4. Use the system tray WAMP menu to select PHP settings>PHP Extensions>Add Extension.
5. In the command-line interface that pops up, type php_apc.dll and press Enter.
6. Using a text editor, open /php/php.ini and add the line apc.rfc1867 = on (it doesn't matter where). If you're trying to test locally and plan to upload large files so you can actually see progress, you'll also want to add the following directives: apc.max_file_size = 200M, upload_max_filesize = 200M, and post_max_size = 200M. Don't do this on a live production server, though, or you're likely to use up bandwidth and disk space allotments, not to mention slowing everyone else down to a crawl.
7. Restart the webserver.

APC should now be set up and initialized. The RFC1867 features of APC — the features that enable you to track file uploads — should now be enabled as an option, and you should be ready to look into our file uploads to enable real-time status.

If you have trouble installing this on windows, just post and I'm sure we can figure it out. In the meantime, I think these steps are self-explanatory enough to get you started.

==========================================================

Testing out the installation

Testing out the installation is pretty simple. Just go into the directory where you initially unpackaged the APC bundle and find the file apc.php. Copy this file into your docroot and open it up in your browser. If the installation worked, you should see a big long explanation about all of APC's features, along with a nice pie chart if the GD library is also installed.

==========================================================

Making A Progress Bar Test

Finally, we're ready to test out this progress bar stuff! It's very freaking easy, and I'm first going to just give you all the code you'll need.


__________________________________________________________


progress.php

<?php
$id = uniqid("");
?>
<html>
<head><title>Upload Example</title>
<script type="text/javascript">
function getProgress() {
CDownloadUrl('get', "getprogress.php?progress_key=<?php echo($id)?>",
function(percent) {
document.getElementById("progressinner").style.width = percent+"%";
if (percent < 100) {
setTimeout("getProgress()", 100);
}
});
}

function CDownloadUrl(method, url, func) {
var httpObj;
var browser = navigator.appName;
if(browser.indexOf("Microsoft") > -1)
httpObj = new ActiveXObject("Microsoft.XMLHTTP");
else
httpObj = new XMLHttpRequest();

httpObj.open(method, url, true);
httpObj.onreadystatechange = function() {
if(httpObj.readyState == 4) {
if(httpObj.status == 200) {
var contenttype = httpObj.getResponseHeader('Content-Type');
if(contenttype.indexOf('xml')>-1) {
func(httpObj.responseXML);
} else {
func(httpObj.responseText);
}
} else {
func('Error: '+httpObj.status);
}
}
};
httpObj.send(null);
}

function startProgress(){
document.getElementById("progressouter").style.display="block";
setTimeout("getProgress()", 1000);
}

</script>
</head>
<body>
<iframe id="theframe" name="theframe" src="upload.php?id=<?php echo($id); ?>" style="border: none;
height: 100px; width: 400px;" > </iframe><br/><br/>
<div id="progress_win"></div>

<div id="progressouter" style="width: 500px; height: 20px; border: 6px solid red; display:none;">
<div id="progressinner" style="position: relative; height: 20px; background-color: purple;
width: 0%;"></div>
</div>

</body>
</html>

__________________________________________________________

upload.php
<?php
$id = $_GET['id'];
?>
<form enctype="multipart/form-data" id="upload_form" action="target.php" method="POST">
<input type="hidden" name="APC_UPLOAD_PROGRESS" id="progress_key" value="<?php echo $id?>"/>
<input type="file" id="test_file" name="test_file"/><br/>
<input onclick="window.parent.startProgress(); return true;" type="submit" value="Upload!"/>
</form>

__________________________________________________________

target.php
<?php
if($_SERVER['REQUEST_METHOD']=='POST') {
$filename = $_FILES["test_file"]["tmp_name"];
$destination = "/path/to/apache22/data/progressbar/uploads/" . $_FILES["test_file"]["name"];
move_uploaded_file($filename, $destination);
echo "<p>File uploaded. Thank you!</p>";
}
?>

__________________________________________________________

getprogress.php
<?php
header('Expires: Mon, 26 Jul 1997 05:00:00 GMT');
header("Last-Modified: " . gmdate("D, d M Y H:i:s") . " GMT");
header('Cache-Control: no-store, no-cache, must-revalidate');
header('Cache-Control: post-check=0, pre-check=0', FALSE);
header('Pragma: no-cache');

if(isset($_GET['progress_key'])) {
$status = apc_fetch('upload_'.$_GET['progress_key']);
echo $status['current']/$status['total']*100;
}
?>
==========================================================

Some Explaining to Do

First, let me say that the above code is direct code examples from IBM, but altered a little bit to make our lives much simpler.

In the first file, progress.php, IBM originally had a Google MAPS API work-around using the GDownloadUrl function. As anyone knows who has tried to use this behind a firewalled web server, you can't validate your KEY (i.e.: <script src="http://maps.google.com/maps?file=api&v=2&key=YOURKEYHERE" type="text/javascript"></script>) without Google being able to access your server. Since this is true, the now awesome GDownloadUrl function is unusable. Fortunately I found the same idea they use for this function already exists in a separate implementation called CDownloadUrl that does not require Google Authentication. As I suspected, it's basically just an XHTTPRequest method that asks for the results of getprogress.php

Also in progress.php there is a bit of confusion as to what exactly the $id value does. Well, the $id value, even though it appears blank, is anything but. It holds a unique ID value, so that we can track our current session without worrying about compromising it with someone else's results. To see it's value, just put "echo $id;" somewhere after the tag to view its result.

Next, is target.php. This file basically just tells our application WHERE to store the new Uploaded file.

Finally, getprogress.php. In this file is one more foreign entry that IBM does not give you: no-caching.

header('Expires: Mon, 26 Jul 1997 05:00:00 GMT');
header("Last-Modified: " . gmdate("D, d M Y H:i:s") . " GMT");
header('Cache-Control: no-store, no-cache, must-revalidate');
header('Cache-Control: post-check=0, pre-check=0', FALSE);
header('Pragma: no-cache');
//....
We HAVE to put the no-cache statement in there for this to work in both Firefox and IE 6+. IE 7 (maybe IE 6, I haven't tested this in that browser yet) CACHES your value. So unless you put this no-cache statement in there, the result you are going to see is the progress bar jump 1 time only (usually around 4-7% on big files or 90% to 100% on smaller files). This is because once IE's Ajax call retrieves the value once, it freaking caches the value and just sits there. And this little annoyance smurf ME OFF (sorry, but it did). I can't believe that not only I have to put hacks in place for IE in css, but now also because it caches VALUES when I don't want it to??????? Sigh........

==========================================================

Conclusion

Run the progress bar example. It works. Works like a freaking dream!! I couldn't believe it but it has finally made me smile once again

It should also be noted here that APC is NOT just for progressive uploads, but instead a caching service. We're basically just using APC's caching ability in order to grant us the nice method of progressive uploads...

Anyways, I hope you enjoyed this tutorial. Take it easy.

==========================================================

Resources
What's new in PHP V5.2, Part 5: Tracking file upload progress
PHP Manaul for APC
CDownloadURL Javascript Function

@author: Jonathon Hibbard


I originally wrote this article in April, 2008, which can be found here.

Friday, December 11, 2009

Simple Optimizing Tips for PHP Part III - Commenting

As we write PHP code (or any language for that matter) we begin to start thinking about optimizing code. But even though our intentions are for "the greater good", we sometimes miss things that should be obvious.

Today's article takes a look at:

Commenting

Commenting, to some, doesn't make sense to be an "optimizing" tip. I'm here to tell you that this is wrong. Commenting is, without a doubt, a key part to optimizing your code!

If we consider all the scripts we have built over the years, one of the fundamental things that allow a script to be a "strong" and "well written" script is its commenting structure.

Comments allow developers to find key methods and components that make a script operate much easier. Comments also allow for maintaining code and, in the process, make debugging these scripts easier. Thus it optimizes not only PHP, but also YOU!

There are 3 supported methods of commenting blocks in PHP. They are:

// (2 forward slashes) - This commenting break comments out one line.

# (or hash) - This commenting break also comments out one line.

The only difference is this method was (and still is) used in Unix

programming.


/* */ - This commenting style is for one to many lines of code.



While knowing the commenting blocks is all good and great, there exists 3 main questions that are asked by all levels of developers:
1) Where should I comment?

2) How often should I comment?

3) What kind of commenting style should I use?

Where should I comment?
This is perhaps the trickest question of all. The key here is if we define where to make comments, we could inadverdently convince a developer to make comments more often then he should. So to answer this quesiton without misleading and causing more harm than good, I have a few examples of "where" one should comment.

You should comment on functions, variables, or complex blocks of code in which:
a) the meaning of the definition is unclear,
b) where it wasn't written in the most optimal way,
c) where it was a "hack",
d) where an "important" piece that solves a complex issue,
e) where a "change" from a privious method existed,
f) when commenting "out" an old method,
g) when setting ground rules for said definitions.

Since there are obviously a lot of instances commenting is great, we must be careful not to over do our comments. Which leads us to the next question...



How often should I comment?
You should not comment too heavily, and you should not comment too lightly. An even medium is suggested, with an influence on the lighter side.

While some will say lots of comments help people understand things, I will disagree and say it causes more complexity than is needed. If we end up having 100 lines of a 300 line file being code, and the other 200 lines of code being comments and whitespace, it very evident that the code is either written so poorly that lots of comments are needed to explain how it works, or that the author himself is not sure how the code works.

Whlie some will say this is not true that they just "like" to make long, elaquent comments, I can respond with that there is a time and place (such as a wiki) for such long-winded explanations.

So, how do we solve this issue of not writing too much, or too little comments?

It's very simple. As stated in the "Where to comment" above, you should mainly be commenting on pieces of code that are not very easy to follow. So, to explain these pieces, here are a few scenarios that should help you on your way:

1) Imagine that you have a variable that has been defined by an included file. A comment where the variable first appears would be an extremely helpful spot to point a new developer (or even your future self!) in the right direction.

2) Say we have a new class or function that we have writte to save ourselves some time. Make a comment explaining the purpose of this class/function!

3) Imagine that you have a few included files that perform other operations that will make the script you're currently viewing operate correctly. You could make 1 comment for all these includes, explaining why they are there and what purpose they have.

4) One day you have to debug a piece of code but you forgot how it works. So you start back-tracking through the code and figure out, piece by piece, everything that was needed to make the end result work. Find the most vital part of your code, and create a comment there explaining just what you found! There may be a workound later.

Whlie these examples are small, the concept should be quite clear. Try to consolidate your comments, but try to keep them short and sweet and to the point! The comments are there to leave breadcrumb trails and help debug, NOT to create a new bestseller book!

What kind of commenting style should I use?
The commenting style I will suggest here is using phpDoc commenting format. To learn about phpDoc, I would recommend their website for more information.

Why phpDoc? Because, for me and the teams I've worked with, the style is clean, organized, and even supported in IDE's such as Zend and Eclipse!

While I know this article was very short, I must break here. A whole book could be focused on the issue of commenting methods (and books DO exist on the subject!), but today just isn't the time for that sort of article. For now, have a merry christmas!

Be sure to check out my next optimizing tutorial : By Example

Thursday, December 3, 2009

How to Fix svn: Directory __DIR__ containing working copy admin area is missing (and other misc SVN shortcuts)

So, you have a directory called myclass, you've added the directory to SVN, and then you issue a commit. During the commit, you get a message saying either a file is outdated, or a directory already exists that causes this commit to fail. No big deal right? So you fix those issues and try to commit again... Only this time, you get a totally different error that can cause a pretty ugly headache when you're trying to solve it.

Problem: svn: Directory 'myclass' containing working copy admin area is missing (and other misc SVN shortcuts)

Solutions
1) You could try and "checkout" the whole project again with the command svn co http://svn.mydomain/my_project/ my_project. This works for some, but in the case above, it may or may not work...

2) You could try and "checkout" only the folder that's having the issues into a temporary directory, then move the .svn folder into the SVN controlled directory where your project lives with the commands:
svn co http://svn.mydomain/my_project/myclass my_temp_folder
mv my_temp_folder/myclass/.svn /path/to/my_project/myclass/

Again, this will work for some, but in the case above it may yet again not solve your issue.

3) Finally, you could copy the folder (in this case, myclass) to a temporary location, force delete it from svn's repository, and copy the folder back in, issue the add, and then commit again with the following commands:
cp -r myclass /home/myuser/myclass_temp/
svn --force delete myclass
cp -r /home/myuser/myclass_temp/ /path/to/my_project/myclass
svn add myclass
svn commit -m "Added myclass folder"

This 3rd step works for me everytime, though the others could work for you as well.


Since we're talking about SVN, here are some helpful commands should you need them...

Checkout a SVN Repository : svn co http://svn.mydomain.com/myrepo/my_project myproject

Force and ADD recursively to all files and directories inside a working SVN folder: svn add --force ./*

UPDATE a SVN project: svn update

Commit a change to your SVN project: svn commit -m "My Message here..."

Delete some files or folders from SVN svn --force delete my_file_or_folder