Thursday, May 24, 2018

Interactive Command Line Directory Picker

Like most developers, I work on multiple projects, and I'm lazy so I don't like to constantly typing in directory paths to change my location in terminal. I used to use a custom bash script to make this task easier but it proved to be hard to maintain and not flexible.

Since I am primarily a PHP developer, I wanted to make a script in PHP to handle the interactive logic bits of determining what project I am trying to navigate to. As an added feature, I wanted to not have to memorize all the possible paths that I've setup in the chooser. To this end, I can enter as little as the function name or the beginning portion of the option or the entire path option:

  • chdir
  • chdir juno
  • chdir juno.api
  • chdir juno.api.logs
If the path entered is the final destination (resulting in a string) then you're redirected to that location. If it's a option (resulting in an array) you're given a choice of where to go.

A simple bash function is created in my ~/.bash_profile. The function takes the first argument it receives and passes it directly to the PHP script. The PHP script then puts the resulting path choice into a temporary file. Then the bash script reads the file. With some additional time spent, I probably could have avoided the temporary file but I didn't feel it was worth it. The PHP script validates the path, so as long as the result in the temporary file is not empty, the bash function changes the current working directory with the cd command.

chdir() {
    echo '' >> /tmp/chdir
    php /Users/david/dev/directory_chooser $1
    route=$(</tmp/chdir)
    rm /tmp/chdir
    if [ "${route}" != '' ]
    then
        cd $route
    fi
}

Because this function is placed into my bash profile script, there is no need to prepend the call with a period so it will result in changing the current working directory. The custom bash script I used previously had to be executed with a dot space prepending it to actually change my working directory.

The bulk of the work happens in a PHP script called directory_chooser. The possible path options are coded into the $routes array. The name displayed is the key. If the value is a string, then it's the destination for that option. If the value of is an array, then it's a deeper set of options. The special case is when an option has no key (index of zero), it's presented as the Root. So choosing juno.api will result in a choice between the Root and logs.

<?php
// Command line colors
$colorReset="\033[0m";           # Text Reset
$colorCyan="\033[0;36m";         # Cyan
$colorBRed="\033[1;31m";         # Bold Red
$colorBGreen="\033[1;32m";       # Bold Green
$colorBCyan="\033[1;36m";        # Bold Cyan

// Available path options
$routes = [
    'juno' => [
        'home'   => '/Users/david/dev/juno-home/Juno-Home',
        'api'    => [
                      '/Users/david/dev/juno-api/Juno-API',
            'logs' => '/Users/david/dev/juno-api/Juno-API/storage/logs'
        ],
        'app'    => '/Users/david/dev/juno-app/Juno-App',
        'mobile' => '/Users/david/dev/juno-mobile/Juno-Mobile'
    ],
    'example' => '/Users/david/dev/example.com'
];

$input = (isset($argv[1])) ? $argv[1] : null;
$route = null;

// If the input is a dot notation value, get the path option desired
if($input !== null && strpos($input, '.') !== false) {
    $parts = explode('.', $input);

    $route = $routes;
    foreach($parts as $part) {
        if(is_array($route) && isset($route[$part])) {
            $route = $route[$part];
        }
    }
}
// If the input is provided but is not a dot notation
elseif($input !== null && isset($routes[$input])) {
    $route = $routes[$input];
} else {
    $route = $routes;
}

// Drill down in the path options until we get to a string path
while(!is_string($route)) {
    $names = array_keys($route);

    if(count($names) == 1) {
        $route = $route[$names[0]];
        continue;
    }

    foreach($names as $index => $name) {
        if($name === 0) {
            echo "{$colorCyan}\t{$index} - Root{$colorReset}\n";
        } else {
            echo "{$colorCyan}\t{$index} - {$name}{$colorReset}\n";
        }
    }

    echo "{$colorBCyan}Which path (numeric index):{$colorReset}\n";
    $handle = fopen('php://stdin', 'r');
    $line = fgets($handle);
    if(isset($names[(int)$line]) && $route[$names[(int)$line]]) {
        $route = $route[$names[(int)$line]];
    }
}

// Make sure path actually exists
// On success, write the path to a temporary file so the bash script can get it
if(file_exists($route)) {
    echo "{$colorBGreen}Changing location to {$route}{$colorReset}\n";
    file_put_contents('/tmp/chdir', $route);
} else {
    echo "{$colorBRed}ERROR: The destination does not exist - $route\n";
}

The while loop makes it so that you can have path options as deep or as shallow as we want.

Because I like a pretty command line tools, I'm colorizing the PHP output with the $colorCyan and similar values. There are libraries to make command line choices like this easy, but I didn't want the overhead of maintaining links to a library, that's why I'm using the basic PHP command line input method of stdin.

The names have been changed to protect the innocent.

Monday, August 5, 2013

How to Securely Communicate with a Non-Secure Server

Security is a very important consideration when building web applications. In order to build trust with your client, they need to know that their information is not going to be compromised. Obviously the best practice is to have the application installed on a server that has a good SSL certificate and the application is only accessible via HTTPS. This is all well and good if you're building an application that will run on a server that you control.

But what if you're building a distributed web application that will be installed by your customers on their servers? I've found that the end clients are, in general, not very savvy when it comes to server setups and they have to be walked through every step. Adding additional requirements or steps to the installation process can just make their head spin. Obviously there are clients out there that are very security conscience and tech savvy and either already have a secure server or know exactly how to set one up; they are much easier to deal with.

Getting down to the nitty gritty, the most important items to protect from snoopers is login credentials. But how do you do this when credentials are being transmitted over a non-secure connection to their server because their server doesn't support HTTPS?

One option is to build your own encryption method. This is rather much of a false sense of security. While the actual information being transmitted is not plain text, it is typically very easy to decipher. I've employed character remapping, usually adding salts, techniques to do this. The two big downfalls of this method are that you are transmitting the key when the page loads and it's a fixed remapping so, for instance, t always equals Y.

Another option is to use a client side encryption schema that can then be decrypted by your application. Javascript plugins, such as SJCL or jscrypto. The problem with any encryption that happens on the client side is that the server has to transmit everything that is needed to encrypt the data, and in turn everything that is needed to decrypt the data. There are ways of getting around the inherent security issues, just look at how Braintree does it. But this is not easy or quick.

The best option if you have a secure server at your disposal, say for client authentication to make sure that your client still should be allowed to use your product, is to send critical communications to that server instead. The principle works like this. Because any communication sent over HTTPS is secure by nature, all you have to do is make sure that all sensitive communications are being routed through your secure server. So in practice, you can have the browser send credentials to the secure server via JSONP (to the HTTPS address) or simple form data. The JSONP is a much more user friendly method in my opinion. The secure server then stores the credentials temporarily (maybe as long as five seconds). Once the secure server has the information, it sends a signal back saying I got the information, you can proceed. The browser then sends a signal to their non-secure server to get the credentials from your secure server. This is done via a cURL request in PHP. Again because you are sending the request to the HTTPS address of your server, the entire communication is encrypted. Your server sends back a json encoded response for instance. So now their non-secure server has the credentials to do what every you need it to do and they were never sent over a non-secure connection.

I wouldn't recommend sending all communications through you server because of the extra traffic, but for login's and changing credentials, the hit wouldn't be that bad.

Your server can either store the credentials in a database or memory via APC. Either way, as soon as their server requests the credentials, wipe them from your server. You don't want to store unencrypted credentials on your server, it's just asking for a disaster.

Monday, July 15, 2013

PHP - Log Maintenance

Running a PHP file from cron is easy enough, simply specify the PHP bin path, the path to the PHP script and where to send the output of the script. The issue comes with how to manage the size of the log file. You can set a job to truncate the file as opposed to append it but that means you loose any of the log lines that were in it. A better option is to use one of the cron jobs to rotate the logs into an archive of some sort.

This script first checks when it was last run, if it's under an hour ago, it skips to the remainder of the job.

We then check the file size, if it's over 1MB it starts the process of archiving it and creating a blank replacement. To be sure we can know when the archived log file was created, we put the epoch time that it was created in the file name.

Every time this portion of the job runs, it also checks to see if there are any log files that are older than seven days.  Because we put the epoch time that the archive was created, this is a simple matter of getting the epoch time back out of the file name and comparing it to our cutoff value. If the file is older than cutoff date, we simply delete it then log that we did so.

// only check log files for maintenance once an hour at most
if($process->lastRan < (time() - 3600)) {
    $logFile = LOG_PATH.'/cron.log';
    // if log file is over 1MB, archive it and create a new one
    if(filesize($logFile) > (1024 * 1024)) {
        // put the epoch time into the name so we can tell when it was archived
        $logFileBk = 'cron.'.time().'.log';
        // duplicate the file with the new name
        copy($logFile, LOG_PATH.'/'.$logFileBk);
        // create a blank 
        file_put_contents($logFile, date('M d, Y H:i:s')."\t[cron_controller]\tTruncated log file - $logFileBk\n");
    }
    // look for any log file that is oder than 7 days
    $end = strtotime('00:00:00 - 7 days');
    $list = scandir(LOG_PATH);
    foreach($list as $i) {
        // get the date date from the file name if it matches the correct form
        if(preg_match('/^cron\.([0-9]{10})\.log$/',$i,$matches) === 1) {
            // if the found date is older than 7 days ago
            if((int)$matches[1] < $end) {
                // delete the file
                unlink(LOG_PATH.'/'.$i);
                echo date('M d, Y H:i:s')."\t[cron_controller]\tDeleted log file - $i\n";
            }
        }
    }
}

This script could be put in either the beginning of the cron job or at the end. But if you are using the PHP Cron Controller that I outlined in this post, I highly recommend doing this log maintenance as early in the script as possible. If you do it at the end of the script, you could run into a situation where a job called from the PHP Cron Controller is working and writing lines when this maintenance is being done.

Monday, July 8, 2013

PHP Logging Class

The last post talked about a PHP Cron Controller, the problem is that because it's a fire and forget controller, it does not capture the output (including errors) of the jobs and do anything with it. That's why we need a way to log the output and the errors to a file for later review.

Since I prefer to do most everything in classes to keep functions and variables isolated,  that's what I did for this set of functions.

The first thing we need to do is to initiate the class. In this function we will be determining the name of the cron job that is to be logged. We can either do this by submitting the job name (process) or it can be determined by the file name that initiated the function. Obviously it's faster to submit the process name, but I like flexibility. If you capture the output of debug_backgrace function, you'll see that it has the file name of the function that called the current function. This is what we use to get the process name if it's not submitted to the class constructor.

The constructor also allows us to change the file that the logs are written into. For safety, we perform some tests on the file specified to make sure it exists and that we can write to it. If either of these tests fail, we use PHP to see if we can fix the problem. We could use shell commands, but my experience is that most servers don't allow PHP to execute shell commands so I'd say, don't bother unless you know you have the ability.

If after all the tests we determine that we can't write to the log file, there is no point in letting the later functions work so we set a flag (writeError) in the class so we can kill other functions immediately.
    public function __construct($process=null, $logFile='cron.log') {
        if($process == null || $process == '') {
            $callers = debug_backtrace();
            $file = $callers[0]['file'];
            $fA = explode('/', $file);
            $file = array_pop($fA);
            $fA = explode('.', $file);
            array_pop($fA);
            if(count($fA) > 1) {
                $this->process = implode('.', $fA);
            } else {
                $this->process = $fA[0];
            }
        } else {
            $this->process = $process;
        }
        
        $this->logFile = LOG_PATH . '/' . $logFile;
        if(!file_exists($this->logFile)) {
            if(file_put_contents($this->logFile, 'Log Created '.date('M d, Y H:i:s')."\n") === false) {
                $this->writeError = true;
            }
        } elseif(!is_writable($this->logFile)) {
            $t2 = @chmod($this->logFile, 0777);
            if($t2 === false) {
                $this->writeError = true;
            }
        }
    }
The class is initiated in the cron job by calling:
$log = new Log_Model('metrics');

Next we want to be able to actually write some useful lines to the log file. The only thing that this function requires is the string to output to the log file. We'll discuss the $die in a little bit. For safety, we perform a very basic test on the input to make sure it is a string and that it's not empty. If the constructor couldn't write to the log file, we don't even attempt to write to the file here, but we do let the other portions of the function work. For standards sake, I like to begin each line with the date and time then the process that this line relates to.

I like to use the file_put_contents function as opposed to using the file handle method because it's simpler. We still have the ability to do the append by adding the FILE_APPEND flag. Since multiple jobs could be writing to the log file at once, we add the LOCK_EX flag.

The die portion of this function has to do with the error handling but it can also be used in the cron job if for instance a database call yielded no results. But since this is less clear in the cron job, I like to implicitly put the exit in the job itself rather than a true in the function call like $log->writeLogLine('Log this line', ture).
    public function writeLogLine($str, $die=false) {
        if(!is_string($str) || $str == '') {
            return false;
        }
        $r = false;
        if($this->writeError == false) {
            $line = date('M d, Y H:i:s')."\t[$this->process]\t$str\n";
            if(file_put_contents($this->logFile, $line, FILE_APPEND | LOCK_EX) !== false) {
                $r = true;
            }
        }
        if($die === true) {
            exit;
        }
        return $r;
    }

The last portion of the class is the error handling. To be fair, this is mostly stolen from the PHP documentation. I've done some minor changes in function where it will send the error output to the log file (via the writeLogLine function) and send the die command to the function if it is a fatal error.
    public function errorOut($errno, $errstr, $errfile, $errline) {
        if (!(error_reporting() & $errno)) {
            // This error code is not included in error_reporting
            return;
        }

        $die = false;
        $str = '';
        switch ($errno) {
            case E_USER_ERROR:
                $str .= "FATAL ERROR [$errno] $errstr in $errfile on line $errline\n";
                $die = true;
                break;

            case E_USER_WARNING:
                $str .= "WARNING [$errno] $errstr in $errfile on line $errline\n";
                break;

            case E_USER_NOTICE:
                $str .= "NOTICE [$errno] $errstr in $errfile on line $errline\n";
                break;

            default:
                $str .= "Unknown error type: [$errno] $errstr in $errfile on line $errline\n";
                break;
        }
        
        $this->writeLogLine($str, $die);

        // if you return false, it will still execute PHP's error handling
        return true;
    }

To enable the error handling, I prefer to explicitly call it in the cron job as opposed to doing it in the class constructor because it's much clearer in the cron job. To do this simply call:
set_error_handler(array($log, 'errorOut'));
The $log is the instance of the class and the errorOut is the function to use in the event of an error.

This method of handling logging and errors works well for jobs that work in the background that you don't typically see the output of. If you also wanted to see the output, it would be easy to add an echo into the writeLogLine function, but for my application, it was completely unnecessary.