How to install phantomjs and casperjs on bluehost linux vps

Last week I needed to automate an internet lookup in real time which required a several step login process (no API available). I was having trouble getting it to work with Curl. I had heard good things about CasperJS and PhantomJS so I figured I would try it out. I decided to use my own bluehost vps server to start with because it is always easier to develop a program which you have full control over.

CasperJS logo

I started with PhantomJS by itself, because I figured if I could get that working I didn’t need casper. However, after attempting the follow the installation instructions and using Phantom directly, I was getting errors that the client class was not found. I tried manually including the class file but that just led to more and more missing class and object errors because the whole library is setup to autoload things. After manually including the vendor/autoload.php file I at least didn’t get any more errors, but the simple examples were giving me responses of 0 so I decided I needed a different approach.

Installing CasperJS was relatively easy, but let me share the steps I followed to actually get phantom installed (it could be there is something I missed which prevented it from working by itself, but since I got casper working later I was satisfied):

  1. Login to the server through SSH (I use putty for this) as root (this is needed later. If you have a non privileged user who can login through ssh then you can start with that. Or if you’ve disabled root logins then login with the user who can become root)
  2. Become a non privileged user (type ‘sudo su [username]’, where username is the owner of your existing web files – the specific user here is important to avoid permission errors later).
  3. Create a directory for casper and phantom, like automation or browsercontrol, in a directory above an existing domain or subdomain so it’s not accessible in a browser (for security reasons)
  4. CD to the new directory and install composer there (even if composer is already a recognized command, do this anyway): curl -s http://getcomposer.org/installer | php
  5. Create a file in that directory called composer.json with these contents (this is straight from the installation guide):
    {
            "require": {
                "jonnyw/php-phantomjs": "4.*"
            },
            "config": {
                "bin-dir": "bin"
            },
            "scripts": {
                "post-install-cmd": [
                    "PhantomInstaller\\Installer::installPhantomJS"
                ],
                "post-update-cmd": [
                    "PhantomInstaller\\Installer::installPhantomJS"
                ]
            }
        }
    
  6. Try this command to install phantomjs:
    php composer.phar install
    
  7. If that doesn’t work (for example there is no bin folder created, and/or phantomjs is not created anywhere locally), then go to http://phantomjs.org/download.html and pick a link to manually download to your server (in your new folder, for example this is the command I used from my new directory):
    wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2
    
  8. Extract the downloaded file (x means extract, j means it’s a bz2 file, and f means the file name is coming next):
    tar xjf phantomjs-2.1.1-linux-x86_64.tar.bz2
    
  9. Use this command to figure out what folders are currently in your path:
    echo $PATH
    
  10. Pick one like /usr/bin which is not for system programs (/bin and /sbin are, so don’t use those to avoid confusion later)
  11. Figure out the absolute path to phantomjs in your system now by finding it and then using pwd to get the path (likely ending with something like phantomjs-2.1.1-linux-x86_64/bin/phantomjs)
  12. Become root (if you logged in as root you can be root again by typing ‘exit’ at the prompt)
  13. Create a symbolic link to phantomjs inside the folder you picked in step 10 (like /usr/bin). Something like this:
    ln -sf /full/path/to/phantomjs /usr/bin/phantomjs
    
  14. Validate it worked by becoming the non-privileged user again and typing “phantomjs –version”. You should see a version number, not a complaint about no command found

Then for CasperJS:

  • Use slightly modified instructions from casperjs.org to install casper from git and put a link to it into your path (as the non privileged user, starting in the new folder you created):
    git clone git://github.com/casperjs/casperjs.git
    cd casperjs
    ln -sf `pwd`/bin/casperjs /usr/bin/casperjs
    

    If you picked a different folder in step 10, use it in the second part of the third command instead of /usr/bin.

  • Validate casper install by typing casperjs from the command line from any folder. You should see info about casperjs, not a complaint about a missing command.
  • In order to use casper within a PHP file, you will need to use the exec command (make sure that is allowed in your php.ini file). Here is a test php file you can use to make sure it is setup fully:

    if(function_exists('exec')) {
    	echo "exec is enabled<br>";
    } else echo "exec is not enabled<br>";
    exec("phantomjs --version", $outArr1);
    exec("casperjs", $outArr2);
    echo "<br>PhantomJS version output:<br>";
    print_r($outArr1);
    echo "<br>CasperJS output:<br>";
    print_r($outArr2);
    

    If you see a version number like 2.1.1 for phantom and a bunch of info about casper (after the CasperJS output line) you are good to go. The next step is to follow the casper instructions to create your javascript logic and then change your casper exec command to be something like this:

    exec("casperjs casperLogic.js --getParam1=\"This is passed to the script\" --getParam2=\"This is also passed to the script\"", $outArr);
    

    Happy automating!

    How to Add a Form in a WordPress Post

    This is a simple step by step guide to creating a form that will capture data and store it in a mysql database on your server from within a wordpress post

    Note: the below example is a live, working form on this page

    Step 1:

    Create a simple form for a petition, contact request, or registration:

    Name:
    Address:
    City:
    State:
    Email:
    Comment:

    Here is the html code for it:

    <table>
    <tr><th>Name:</th><td><input id="userName" /></td></tr>
    <tr><th>Address:</th><td> <input id="userAddress" /></td></tr>
    <tr><th>City:</th><td> <input id="userCity" /></td></tr>
    <tr><th>State:</th><td> <input id="userState> /></td></tr>
    <tr><th>Zip:</th><td> <input id="userZip" /></td></tr>
    <tr><th>Email:</th><td> <input id="userEmail" /></td></tr>
    <tr><th>Comment:</th><td> <textarea id="userComment" ></textarea></td></tr>
    <tr><th><td colspan=2><button type="button" onclick="saveUser();">Save</button></td></tr>
    </table>
    

    Step 2:

    Create a javascript function to process the form and send it to the server. The following is a javascript function added to the page using the CSS & Javascript Toolbox plugin (it uses jquery because wordpress already has that available):

    //Process the form and save the data record
    function saveUser() {
      //First gather the form parameters and make sure name and email at least are populated
      var name = jQuery("#userName").val();
      var address = jQuery("#userAddress").val();
      var city = jQuery("#userCity").val();
      var state = jQuery("#userState").val();
      var zip = jQuery("#userZip").val();
      var email = jQuery("#userEmail").val();
      var comment = jQuery("#userComment").val();
      if (name.length<1 || email.length<1) {
        alert("Please at least enter your name and email.");
        return false;
      } else {
        //Now send the data to a server side function to really validate it and save it.
        jQuery.ajax({
          type: "POST",
          url: "/ajax/saveUser.php",
          data: { name:name,address:address,city:city,state:state,zip:zip,email:email,comment:comment }
        }).done(function( results ) {
          if(results.length<1){ // network error
            alert("There was a network error, please try again or contact support and tell them what you are trying to do.");
          } else { // this is a successful interaction
            var resultObj = jQuery.parseJSON(results);
                    
            if (resultObj.errorMsg.length>0) {  
              alert(resultObj.errorMsg);
            } else {
        	  //Record save successful
              alert("Thanks for your information, it was saved successfully!");
              //Show the user what they have entered:
              jQuery("#userList").html(resultObj.userList);
            }
          }
        });
      }
    }
    

    Step 3:

    Make sure you have a database table ready to store the information. Below is a simple table used to store the info in this example:

    CREATE TABLE IF NOT EXISTS `user_info` (
      `userID` int(11) NOT NULL AUTO_INCREMENT,
      `userName` varchar(80) DEFAULT NULL,
      `address` varchar(100) DEFAULT NULL,
      `city` varchar(80) DEFAULT NULL,
      `state` varchar(40) DEFAULT NULL,
      `zip` varchar(5) DEFAULT NULL,
      `email` varchar(80) DEFAULT NULL,
      `comment` text DEFAULT NULL,
      `userIP` VARCHAR( 30 ) DEFAULT NULL,
      `dateAdded` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
      PRIMARY KEY (`userID`),
      KEY `userIP` (`userIP`)
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;
    

    Step 4:

    Create the server side code to accept the form data, really validate it, and put it in the database or send back an error. Here is the contents of a server side code which will do all that for the example data above. Since this form is “in the wild”, I’m capturing the IP address because I will only show results to people that they entered themselves:

    /*
     * File: saveUser.php
     * Description: This file takes the passed in user information and validates it before
     * 	saving it to the database and returning content to show on the page. 
     * Inputs (in POST): 
     * 	userName, address, city, state, zip, email, comment
     * Outputs:
     * 	either an error or nothing
     */
    include("db.php"); // define your database credentials in another file
    
    #connect to the database
    $db = mysqli_connect($servername, $username, $password, $dbname);
    mysqli_select_db($db,$dbname) or die('Could not select database');
    
    #Get the passed in parameters into local variables, escaped for database input
    $userName = empty($_POST['userName'])?'':addslashes($_POST['userName']);
    $address = empty($_POST['address'])?'':addslashes($_POST['address']);
    $city = empty($_POST['city'])?'':addslashes($_POST['city']);
    $state = empty($_POST['state'])?'':addslashes($_POST['state']);
    //only accept 5 numbers for zip
    $zip = empty($_POST['zip'])?'':substr(preg_replace('/\D/','',$_POST['zip']),0,5);
    $email = empty($_POST['email'])?'':addslashes($_POST['email']);
    $comment = empty($_POST['comment'])?'':addslashes($_POST['comment']);
    $userIP = $_SERVER['REMOTE_ADDR'];
    
    #This is an array used for gathering all the outputs of this file
    $jsonObj = array();
    $jsonObj['errorMsg'] = "";
    $debug=false;
    
    #Validate inputs
    if (empty($userName) or empty($email)) {
    	$jsonObj['errorMsg'] = "Please at least enter your name and email.";
    } else if (strpos($email,'@')===false) {
    	//there are many more validations that can be made for emails but this is a start
    	$jsonObj['errorMsg'] = "Please enter a valid email.";
    }
    
    #Enter the data record
    if (empty($jsonObj['errorMsg'])) {	
    	$sql = "insert into user_info (userName,address,city,state,zip,email,comment,userIP) 
    			values
    			('".$userName."','".$address."','".$city."','".$state."','".$zip."','".$email."',
    			'".$comment."','".$userIP."')";
    	if ($debug) echo "about to run $sql ";
    	mysqli_query($db, $sql); $err = mysqli_error($db); if (!empty($err)) $jsonObj['errorMsg'] = $err;
    }
    
    #Now get the list of data entered by this IP, without any script tags to prevent XSS:
    $jsonObj['userList']='';
    if (empty($jsonObj['errorMsg'])) {	
    	$sql = "select userName,city,state from user_info where userIP='".$userIP."'";
    	$rs = mysqli_query($db, $sql); $err = mysqli_error($db); if (!empty($err)) $jsonObj['errorMsg'] = $err;
    	while($row = mysqli_fetch_assoc($rs)) {
    		$jsonObj['userList'].= "<tr><td>".strip_tags($row['userName'])."</td><td>".strip_tags($row['city']).
    				"</td><td>".strip_tags($row['state'])."</td></tr>";
    	}
    }
    
    #Now send back the data in json format
    echo json_encode($jsonObj);
    

    Step 5:

    Do something with the results! You could export them, feed them into another system using an API, send them in an email to someone, or just display them like so:

    Name City State
    You have not entered any user records yet, try it out!

    How to use Ajax in a WordPress Post

    I recently put together an Ajax autocomplete example using PHP and Mysql so I could post it on my blog, but after getting it working outside wordpress on an independent page I had trouble getting it into wordpress.

    I had used plugins in the past which allowed for php code to be used with shortcodes, but inline javascript calls are not something I had gotten past the wordpress editor before. The php code used for retrieving data from the database is in a separate file so no shortcode is needed for that portion.

    The inclusion of the javascript autosuggest functions is something I first tried putting in a code block in the wordpress post itself as a link to the external file (autosuggest.js), but unfortunately that didn’t work. Then I saw an article where someone suggested putting the javascript in the theme file header.php and surround it with a php if statement checking for the specific post you want it to appear on. I tried that but later removed it because I wanted my ajax to work on the index page, archive pages, and category/tag/author pages – not just the “single post” page.

    Then I went back to the plugin I had installed (CSS & JavaScript Toolbox) for css and javascript and tried putting the entire contents of the autosuggest.js file into one javascript block and installing that in the footer. That worked perfectly so I did the same thing for the css file using the same plugin.

    The next challenge was to add the inline JavaScript (ajax) function calls from within the html code for the sample input box in the wordpress post. When I tried just putting it all into the html editor for wordpress the javascript was removed. This was not just a snippet of complete javascript or css though, so I couldn’t just use the same plugin to make another code block and put it in the header or footer.

    Since PHP is executed as the page loads, I figured I could put my auto-complete input box with ajax call into a php shortcode and put the shortcode where I wanted the input box to be in my html code. That way wordpress would not filter out the javascript when saving the post contents to the database because it would be hidden behind a shortcode which would be replaced during page loading.

    This worked beatifully, including the inline javascript by putting it inside a php shortcode. I used the Shortcode Exec PHP plugin for this, which is very easy to work with.

    Simple Ajax Auto-Complete Example

    In a previous post I wrote about how I learned ajax quickly, but recently I learned that the main link I learned from is no longer valid. So I decided to make a simple example I could host myself. Along the way I learned that it is not trivial to include ajax within a wordpress page, but I wrote another post about how I figured that out. 🙂

    The input box below is enabled with auto-complete using a simple ajax example. The javascript can be seen here, the css can be seen here, and the php ajax function can be seen here. The input box below calls the getScriptPage javascript function if there are at least 2 characters present. That javascript function calls the php ajax function using a server request. The text sent back from the server by the php file is then handled by the handleHttpResponse javascript function, which just populates a hidden div with choices. The getScriptPage function makes that hidden div (with an id of “box”) visible if the response from the php ajax call is not empty.

    The database table being queried has 1000 rows in it, in the following format: name100 (where 100 is the number of the record, 1-1000). So if you start typing “name” in the box, after na you will start seeing 5 suggestions below. If you keep typing the suggestions will be updated as you type.

    HTML code for input box

    <div id="headersearch">
    	<div id="searchbar">
    		<div class="ajax-div">
    			<div class="input-div">
    			<input type="text" autocomplete="off" onKeyUp="if(this.value.length>1) 
    				getScriptPage('box','searchTerm','http://www.wilycode.com/'); else box(0);" 
    				name="searchTerm" value="Enter Name Here" onClick="if(this.value.substr(0,6)=='Enter ') this.value=''"
    				id="searchTerm" onBlur="if(this.value.length<1) this.value='Enter Name Here'">
    			</div>
    			<div id="box"></div>
    		</div>
    	</div>
    </div>
    

    April 24th 2014 Update:

    I was asked to add arrow navigation to an autosuggestion box I created this week using the method described above, and found that there are easier ways to do this. I found that jquery UI has an autocomplete widget which can be used to enable arrow key navigation among many other features. However I had trouble getting the examples at api.jqueryui.com/autocomplete/#entry-examples to work, so I did some more searching and found a step by step implementation using a php file for the source element of the autocomplete function which I was able to get working pretty easily, at www.pontikis.net/blog/jquery-ui-autocomplete-step-by-step.

    The Essence of AJAX – How I learned it in an hour

    originally posted at http://www.goodfeelingplace.com/the-essence-of-ajax-how-i-learned-it-in-an-hour
    I was a bit resistant to learning ajax for a couple years because I didn’t see a need for it in the programs I was developing.  However, one day I learned it without knowing it.  I was working with code written by someone else which called a javascript function that invoked a url, then called another function to handle the results of the url.  An example that looks very similar to the code I was working with is at http://www.captain.at/howto-ajax-form-post-request.php (unfortunately that page no longer exists, but I’ve made my own example that you can see at http://www.wilycode.com/simple-ajax-auto-complete-example/ – the details below refer to the not found page but the concepts are the same).

    If you look at that code, there are 3 javascript functions used – get (this is called from clicking on a button in the form on the page), makePostRequest (this is the function which calls the url and sends parameters to it via the post method), and alertContents (this function is called from the makePostRequest function when the response is received from the url).

    The get function just creates a string with parameters to pass, defines an url, and then passes them to the makePostRequest function.  Here are the main lines of code in the makePostRequest function and what they mean:

    • http_request.onreadystatechange = alertContents; – this says when the http_request.readyState changes, then call the alertContents function.
    • http_request.open(‘POST’, url, true); – this defines what url is going to be called and what method is going to be used
    • http_request.send(parameters); – this sends the parameters specified in the post array to the url requested

    There are a few other lines specifying a the header of the request, and a bunch of lines in the beginning that setup the http_request object to work for different browsers.

    Then in the alertContents function (which is actually called several times, whenever the readyState changes), once the readyState of 4 is reached (you can read more about readyStates at http://www.yaldex.com/wAjax/DiggingdeeperintoHTTPreadystates.html) the http status is checked.  If the status is 200 (and not 404, 501, 301, etc. – see http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for a full list with explanations) then the result of that http request (like what you would get if you put that url into a browser address bar and sent the appropriate post parameters in the header – done through a script, not a browser – then checked the page source) is stored into a javascript variable and you can do whatever you want with it.

    In that example they just stick it into a span element on the existing page, but you can do many other things.  If you invoke an url which takes as parameters the values of certain form elements on the page, you could return a list of values to populate another dropdown with, etc.  So you can take the example and modify the parameters passed, what the invoked url does, and what is done with the results – then you will have your own ajax function!

    How to allow up and down arrows to navigate your form fields

    originally posted at http://www.goodfeelingplace.com/how-to-allow-up-and-down-arrows-to-navigate-your-form-fields
    I learned something pretty cool last week. I fielded a request from a client who wanted to be able to use the up and down arrows on his keyboard to navigate a rather large form on the site I created for him. I believed it was possible but I had never done it. So after doing some research I figured out how to do it. Below is a simplistic version (put your cursor in one of the text boxes below and push the up or down arrow keys to move between them):

    1-1 1-2 1-3
    2-1 2-2 2-3
    3-1 3-2 3-3

    The HTML for the simple table is as follows:

    <table border="1">
    <tr>
    <td>1-1</td>
    <td><input id="ItemCost11" onkeydown="return checkKey(event,3,3,'ItemCost',1,1)" maxlength="30" name="ItemCost11" size="3" /></td>
    <td>1-2</td>
    <td><input id="ItemCost12" onkeydown="return checkKey(event,3,3,'ItemCost',1,2)" maxlength="30" name="ItemCost12" size="3" /></td>
    <td>1-3</td>
    <td><input id="ItemCost13" onkeydown="return checkKey(event,3,3,'ItemCost',1,3)" maxlength="30" name="ItemCost13" size="3" /></td>
    </tr>
    <tr>
    <td>2-1</td>
    <td><input id="ItemCost21" onkeydown="return checkKey(event,3,3,'ItemCost',2,1)" maxlength="30" name="ItemCost21" size="3" /></td>
    <td>2-2</td>
    <td><input id="ItemCost22" onkeydown="return checkKey(event,3,3,'ItemCost',2,2)" maxlength="30" name="ItemCost22" size="3" /></td>
    <td>2-3</td>
    <td><input id="ItemCost23" onkeydown="return checkKey(event,3,3,'ItemCost',2,3)" maxlength="30" name="ItemCost23" size="3" /></td>
    </tr>
    <tr>
    <td>3-1</td>
    <td><input id="ItemCost31" onkeydown="return checkKey(event,3,3,'ItemCost',3,1)" maxlength="30" name="ItemCost31" size="3" /></td>
    <td>3-2</td>
    <td><input id="ItemCost32" onkeydown="return checkKey(event,3,3,'ItemCost',3,2)" maxlength="30" name="ItemCost32" size="3" /></td>
    <td>3-3</td>
    <td><input id="ItemCost33" onkeydown="return checkKey(event,3,3,'ItemCost',3,3)" maxlength="30" name="ItemCost33" size="3" /></td>
    </tr>
    </table>
    

    The javascript function used is as follows:

    <script type="text/javascript"><!--
    function checkKey(event, numRecs, numCols, myName, myRow, myCol) {
      var key = event.keyCode;
      var bSetFocus = false;
      var myId = '';
    // if the left arrow has been pressed and we're not in the first column
      if( key == 37) {
      if (myCol-1 > 0) {
      myId = myName + myRow + (myCol-1);
      bSetFocus = true;
      }
      }
      // if the up arrow has been pressed and we're not in the first row
      if( key == 38) {
      if (myRow-1 > 0) {
      myId = myName + (myRow-1) + myCol;
      bSetFocus = true;
      }
      }
      // if the right arrow has been pressed and we're not in the last column
      if( key == 39) {
      if (myCol+1 <= numRecs) {
      myId = myName + myRow + (myCol+1);
      bSetFocus = true;
      }
      }
      // if the down arrow has been pressed and we're not in the last row
      else if( key == 40 ) {
      if (myRow+1 <= numCols) {
      myId = myName + (myRow+1) + myCol;
      bSetFocus = true;
      }
      }
      if (bSetFocus) document.getElementById(myId).focus();
    } // end checkKey function
    // --></script>
    

    In my code I generate the HTML using PHP so when I pass the number of records to the checkKey function (3 in this example) it is the number of rows returned from a database query, and the current row for any given row of the table is the counter in a loop I use to print the table.

    By the way, if you want to allow for the left and right arrows to move left and right in a form, you can do that by expanding the checkKey function to look for key code 37 (left arrow) and key code 39 (right arrow), and you’ll need to implement a column ordering scheme (like the counter I used for the rows) so you’ll know where to take the cursor.

    Four Direction Code

    (submitted by Mr Arrows below, but the comment fields do not accept code so I put it in the main post. I actually had trouble putting two working code samples in the same post so I just updated the main example above to include the left and right arrows)

    What if there are values in each field?

    These two javascript functions need to be defined, which will tell you the index of the start of the current selection and the end of the current selection (which will be the same if nothing is selected and you just have a blinking cursor in the input box). They both take as input the same object (called o below) that is passed into checkKey as the last param (called myObj there):

    function getSelectionStart(o) {
      if (o.createTextRange) {
      var r = document.selection.createRange().duplicate();
      r.moveEnd('character', o.value.length);
      if (r.text == '') return o.value.length;
      return o.value.lastIndexOf(r.text);
      } else return o.selectionStart;
    }
    function getSelectionEnd(o) {
      if (o.createTextRange) {
      var r = document.selection.createRange().duplicate();
      r.moveStart('character', -o.value.length);
      return r.text.length;
      } else return o.selectionEnd;
    }
    

    Then you need to define some variables inside the checkKey function, to get the current input value and the index of the cursor position within that value:

      var inputVal = myObj.value,
      startIndex = getSelectionStart(myObj);
      endIndex = getSelectionEnd(myObj);
    

    I only use startIndex because it will be the same as endIndex when nothing is highlighted or selected in the input box.

    Next you can use a simple regular expression (or any other string function you want) to see if there is any text before (for the left arrow condition) or after (for the right arrow condition) the cursor. For example, this is the code I use when when the left arrow has been pressed:

      var testBC = inputVal.substring(0, startIndex);
      var blankRE=/^\s*$/;
      //alert("beforeCursor is '"+testBC+"');
      // if the regular expression test passes, then there is nothing to the left and we can move cells
      if (blankRE.test(testBC)) {
      var nextField = document.getElementById(columns[colIndex-1]+String(myRow));
      nextField.focus();
      //If we are moving into a select dropdown, restore the previously selected value half a second after it switches to the first or last option (I don't know how to prevent that, so this is my workaround):
      if (nextField.tagName=="SELECT") {
      var cur_ind = nextField.options.selectedIndex;
      var cur_val = nextField.options[nextField.options.selectedIndex].value;
      var cur_text = nextField.options[nextField.options.selectedIndex].text;
      setTimeout(function(){nextField.options.selectedIndex=cur_ind}, 5);
      }
      return false;
      }