Mark Foster's Blog

Category: Development

File upload services for developers
This is a listing of file upload services for developers. They typically provide a front-end JavaScript module to embed the uploader on your site and a back-end service to receive the uploads. Some will also provide workflow and image processing services
- https://www.filestack.com/
  - URL based image transforms including crop, resize, compress, etc.
  - AI tagging, object regonition, OCR
  - Store files on multiple services including S3
  - Upload modal or dropzone
  - Upload from device, Link, web search, Facebook, Instagram, Google Drive, Dropbox, etc.
- https://transloadit.com/
  - Image transforms including crop, rotate, resize, etc
  - Import/export to multiple services including S3
  - Face detection
  - Custom workflows using JSON
  - tus.io protocol to resume uploads
  - Uses and created Uppy as the client side uploader
- https://cloudinary.com/
  - URL based transforms including crop, resize, rotate, text, shadow, etc.
  - Face detection
  - Upload modal or dropzone
  - Upload notifications
  - Third party upload sources including Dropbox, Facebook, Instagram
  - Can store to S3
  - Workflows
- https://uploadcare.com/
  - Image operations including sharpen, crop, size, rotating, filters, etc.
  - Face recognition
  - Custom storage options including S3
  - Select file or drag and drop
  - Upload widget configuration page
- https://sendtomycloud.com/
  - Use your cloud storage
  - Minimal upload widget
- https://ospry.io/
  - Basic upload service
Alternatively, you can use one of these JavaScript modules and build your own back-end to handle uploads or use AWS S3 with a Gateway-Lambda setup to sign upload requests.
- https://uppy.io/
- https://fineuploader.com/
July 29, 2018
PHP/Apache running on Linux won’t connect to a PostgreSQL server
SELinux will block PHP/Apache from connecting to PostgreSQL (and probably any other DB) by default on some Linux distributions. If you are trying to get PHP to connect to a PostreSQL DB on a linux box for the first time and you are sure your pg_hba.conf on the target box is setup correctly then try this:
```
setsebool -P httpd_can_network_connect 1
```
This should configure SELinux to allow Apache/PHP to connect to other hosts.
January 25, 2012
National Broadband Map Review
The National Telecommunications and Information Administration (NTIA) in collaboration with the FCC has published a series of broadband maps on a new site called National Broadband Map (NBM). These maps show what broadband services are available throughout the United States as well as other interesting broadband data.

Hit this link and click the “Explorer the Map” option on their main page to see a map of the US with shaded areas where selected broadband services are available. You can click different selections above the map to toggle the various broadband technologies. To see other maps such as advertised versus actual broadband speeds click on the “Show Gallery” option in the lower right hand corner.

Rochester, NY does pretty well on advertised versus actual although there a few slower than advertised points here and there. Upload performance data is also available. Usually the cable and DSL providers don’t brag much about upload performance likely because in most cases it is lousy compared to download performance. I think upload performance will become more important to the typical internet user than it as in the past now that people are sharing their pictures and video online.

The NBM site use a variety of open source technologies including:
- JQuery – My favorite JavaScript library.
- Modernizr – A JavaScript library to detect browser capabilities.
- OpenLayers – Provides a JavaScript API to display WFS and WMS GIS layers.
- GeoServer – A Java based server software that provides WFS and WMS services.
What is particularly interesting about the site is the developer resources. They provide a series of API’s you can call from your own web applications to use their data. Output formats include XML, JSON, and JSONP implementations. If you want to use the data locally without the APIs you can download it.

I do have a couple criticisms regarding the maps and ironically, they are bandwidth related. The first is that there are too many tiles returned when viewing the default map of the US. I noticed the map was a little slow to fill in. When I enabled Firebug and clicked on the “Explore the Map” option off the main page, over 500 tiles were pulled down. In fact, Firefox/Firebug became unresponsive. I would expect less than 30 256×256 tiles need to be pulled down for a reasonably sized browser window. I wager there is something goofy going on like a bounding box not set for the area displayed.

My second criticism is that the site is not using gzip to compress JavaScript files. Modern web applications tend to lay on the JavaScript pretty heavy and this one is no exception. OpenLayers.js is nearly 1MB all by itself. By enabling gzip on sites with large JavaScript files you can significantly improve site performance. This is a good topic for a future post.

Overall I think the National Broadband Map Site is an excellent resource. It provides very useful data on broadband technologies/speeds, makes this data available via APIs or download, and also demonstrates a variety of open source web application technologies.

Is it worth the $20 million that contractors were paid to build the map? I would say certainly not at first glance but I would want to hear the whole story before I jump to conclusions. I.e. how much of that $20 million was spent on actual development? I am much more skeptical of the alleged $293 million required to collect the data.
February 24, 2011
Titanium Appcelerator Quirks (Updated)

I recently started tinkering with the Appcelerator Titanium Development Platform and have run into a couple of quirks.

Web Inspector will not Launch

If you create a Titanium project on any drive other than C:\ on a Windows system, you will not be able to launch the Web Inspector. This is a apparently a known bug in Titanium Developer.

Conflicting PHP Installations

If you attempt to launch a Titanium Appcelerator PHP app in Titanium Developer on a Windows system that already has a local PHP installation, you may see the following error:

Kroll Event Window Instance: YourApp.exe – System Error

The program can’t start because php_mbstring.dll is missing from your computer. Try reinstalling the program to fix this problem.

Don’t bother re-installing anything. What is likely happening is that your pre-existing PHP installation added a “PHPRC” environment variable that points to your existing PHP install folder. I think Titanium’s command line PHP instance sees this variable and then attempts to use the php.ini file in that path. Your existing PHP installation folder does not likely have the PHP settings and/or module paths that Titanium needs so it breaks.

To fix this without renaming your current PHP installation folder you can delete the “PHPRC” environment variable. Your existing PHP installation should work fine without it because PHP will look for the php.ini file in the same folder it is running from. To delete the “PHPRC” environment variable on Windows 7, click Windows Start, right click “Computer”, left click “Properties” -> “Advanced system settings” -> “Environment Variables…” -> select “PHPRC” under “System variables”, and then click “Delete”.

Hopefully you can now launch your Titanium application without an error and your existing PHP installation will continue to work normally.

January 10, 2011
Php-cgi.exe application error on IIS with FastCGI
I recently installed PHP 5.2.14 on a Windows 2003 machine running IIS 6 with FastCGI to do some PHP testing at home. I chose the fast CGI install and added several extensions during the install including Curl and Oracle. When I attempted pull a test page after the install completed, I saw the following php-cgi.exe application error on the Windows server’s desktop:

The instruction at “0x100f36ec” referenced memory at “0x000c0194”. The memory could not be “read”.

IIS sent the following error message back to the requesting browser after a couple minutes:

FastCGI Error
The FastCGI Handler was unable to process the request.

Error Details:

* The FastCGI process exited unexpectedly
* Error Number: -1073741819 (0xc0000005).
* Error Description: Unknown Error

HTTP Error 500 – Server Error.
Internet Information Services (IIS)

After some trial and error I was able to get my test page to display if I commented the “extension=php_curl.dll” and “extension=php_oci8.dll” lines in my php.ini file:
```
; Local Variables:
; tab-width: 4
; End:
[PHP_CURL]
;extension=php_curl.dll
[PHP_GD2]
extension=php_gd2.dll
[PHP_MSQL]
extension=php_msql.dll
[PHP_MSSQL]
extension=php_mssql.dll
[PHP_MYSQL]
extension=php_mysql.dll
[PHP_MYSQLI]
extension=php_mysqli.dll
[PHP_OCI8]
;extension=php_oci8.dll
[PHP_PDO]
extension=php_pdo.dll
[PHP_PGSQL]
extension=php_pgsql.dll
[PHP_SHMOP]
extension=php_shmop.dll
[PHP_SOAP]
extension=php_soap.dll
[PHP_SQLITE]
extension=php_sqlite.dll
[PHP_XMLRPC]
extension=php_xmlrpc.dll
```
After some more trial error I was unable to get PHP to work without leaving the two lines commented. I tried both the VC6 thread and VC6 non thread safe versions and both exhibited the same behavior. On the PHP download page there is a “Which version do I choose?” section that basically explains that I should be using the VC9 version for IIS. Unfortunately, I only saw the PHP 5.3.3 VC9 download and I wanted to test with PHP 5.2.14.

I downloaded PHP 5.3.3 anyway and it worked. I guess I will be testing with PHP 5.3.3.

On a somewhat related note, if you are using FastCGI with IIS, you will probably want the VC9 PHP 3.3.3 non thread safe version. This article explains why.
October 11, 2010
How to get the contents of an Oracle CLOB data field in PHP
The Oracle “CLOB” (Character Large Object) is a data type used to store up to 4 Gigabytes of text. Retrieving the contents of a CLOB is not as intuitive as you might think.

Let’s say you have a CLOB field/column named “mychars” in an Oracle DB table named “mytable” along with some other fields. You want to simply echo out the text in the “mychars” field:
The above code will give you an error that looks like the following:

Catchable fatal error: Object of class OCI-Lob could not be converted to string in somefile.php on line 14

If you try to do a print_r() on the CLOB in an attempt to figure out what you are dealing with you will get something that looks like:

OCI-Lob Object ( [descriptor] => Resource id #3 )

This is because a Lob object is returned instead of the contents of the CLOB.

To get the CLOB contents you will need to call the load() or read() methods on the returned object. The latter will require the length of data to read in bytes but has the advantage of not being limited by the script memory limit:
```
load();
            //or
            echo $row['mychars']->read(2000);
        }
    }
?>
```
September 22, 2010
Must-have web application development tools
I have come up with a list of my “must-have” development tools:
1. Dual Monitors – Developing with two monitors will make you much more productive simply because you spend less time switching between windows all day if nothing else. Monitors are pretty cheap and the productivity gains will more than pay for a dual output video card and second monitor. Even Microsoft says so.
2. Firebug – If you do any kind of web development you should have Firebug on your tool belt. This Firefox Add-on will tell you exactly what CSS properties are being applied to an HTML element and from where and then allow you to change those properties on the fly in the browser. It also offers JavaScript debugging, a DOM tree inspector, and last but not, least, a “Net” panel that allows you to see all your browser requests, responses, and times.
3. IE Developer Toolbar – IE’s answer to Firebug. It is not quite as full featured as Firebug in my opinion but it does at least allow you to inspect an element and determine how styles are being applied to it and where. This is quite useful since IE has a different box model than Firefox and you can use all the help you can get when trying to make a site look the same in both browsers.
4. Charles Web Debugging Proxy Application – Charles acts as an intervening proxy to your web browser that records all the requests and responses. Some of this functionality overlaps with Firebug but Charles goes a bit further by providing request breakpoints, request editing, throttling, and DNS spoofing.
5. Firefox Web Developer Toolbar – This is another very useful Firefox Add-on that allows you to:
  - Display element attributes in-line with the page you are viewing.
  - View a page’s JavaScript generated HTML.
  - Resize the browser window to preset sizes.
  - Outline different types of elements in the page.
  - Quickly disable, enable, and delete cookies
  - The list goes on…
6. JQuery – JQuery is a JavaScript library but I also consider it an important tool to make JavaScript programming less painful. It allows you to easily select a DOM element you want to manipulate without typing a whole lot and handles many of the browser idiosyncrasies. Using JQuery’s selectors, you can easily change attributes and chain those changes to together. JQuery also has shortcuts for event handling, effects, AJAX, rich UI components, and anything else that is repetitious, boring, or aggravating to do in plain JavaScript. It takes a little time to learn JQuery but it quickly pays off. While there are many other JavaScript libraries available, I would say JQuery has become the most popular and so does this very scientific poll ;). If JQuery or one of its hundreds of plug-ins don’t do what you need then JQuery will work very well along side some of the other popular JavaScript libraries such as Dojo, Prototype.js, Ext.js, Mootools, and YUI.
7. Putty – If you need to connect to your web host via Telnet or preferably SSH, Putty is tough to beat for the price.
8. WinSCP – WinSCP provides SSH file transfer for Windows machines. If you use Windows on your desktop and a Linux host, this is one of the best ways to upload your files.
9. Notepad++ – If you want a basic, lightweight text editor to do your coding with color syntax highlighting Notepad++ is a great choice. Even if you use a full featured IDE for your development I find it handy to have a good text editor handy. Notepad++ has several plug-ins available as well.
10. PHPUnit – If you are developing a web application that you think will be around for any length of time then unit testing is a critical time saver. Chances are you already write tests to check if your code is working if you don’t have a user interface built yet so often you are already committing the time to writing tests. Unit tests developed using a unit test framework allow you to make “assertions” regarding the output of each function or small “units” of code. Unit tests stick with your code throughout it’s life cycle and are usually executed before you check-in a new change. When you run your unit tests using the unit test framework’s executable you can generally test just a single class or your entire code base. This will quickly tell you if the code you just wrote is working correctly and if you have inadvertently broke any existing code elsewhere. For current job my team and I code in primarily PHP and JavaScript so we use the PHPUnit and JsUnit frameworks but there are unit testing frameworks available for nearly all popular programming languages including C++, Java, C#, etc.
11. Selenium – While unit testing covers individual units of code, integration tests cover how everything works together. Selenium accomplishes this by allowing you to build scripts that you can playback to emulate a user’s browser interacting with your application. Like unit tests, you can make assertions that elements in the web interface are working the way they should. This allows you to effectively perform an automated regression test of your application so you can make sure your code changes didn’t break any interactions between the components of the application. Although I personally really like Selenium, there are other good automated test tools such as Watir
12. Web Application Vulnerability Scanning Software? – This is an area that will strongly depend on your budget. I think vulnerability testing is essential but short of doing a lot manual probing and experimenting, it is going to cost you. Even if you have read the OWASP guide back and forth and are careful to escape all your inputs, I still think it is important to run a test tool against your app before you release it into the wild. Although automated testing won’t reveal all your security issues it will at least reveal some of the more embarrassing ones. If nothing else, it is important that the script kiddies don’t find anything interesting when they do the same. While there are many open source security tools, I am not aware of any that will do automated application security scanning like HP’s WebInspect, IBM’s Rational AppScan, or Acunetix WVS. I found an interesting comparison between the three here. I would be interested in any alternatives if you know of any.
13. Security Web Sites – The following web sites are good security “tools”:
14. WebLOAD? – Instead of hoping your web application will hold up to high traffic volumes, wouldn’t you like to really know for sure? A load testing tool is essential to predicting how your application will behave under load and will also help identify bottlenecks in the application that can be optimized. Optimizing your application based on load test data will help to ensure you are focusing your optimization efforts on the real problem areas. In my current position we were using the alleged open source version of WebLOAD, which, worked pretty well. Unfortunately Radview is making new versions, err… I mean, pro, closed source and I am guessing the licensing costs are ridiculous just by the fact that they don’t list the price on their website or provide a shopping cart so you have to contact their sales folks (I am not a big fan of this practice). I am fine with convincing my boss to buy software if need be so long as it doesn’t cost so much he will laugh at me. So… I am in interested to hear about any other load test tools you have experience with.
15. Subversion – Version control is essential if you are working on a project for any length of time and especially with other people. Subversion is easy to setup and as long as you back up your Subversion server and make frequent commits, your code will be safe and you can always revert to a previous version if you really mess it up. CVS is also an option but I prefer Subversion because it maintains versions across the entire code base instead of just individual files. This allows you to easily revert a bad multi-file commit. There are plenty of clients available for Subversion and many development tools have subversion support built-in or provide plug-in options.
Well that’s it for this post. I would really like to hear about any other must-have web app dev tools that you think should be on the list.
September 9, 2009
25 ways to insecurity

The 2009 CWE/SANS Top 25 Most Dangerous Programming Errors was recently released by CWE/SANS.

Most of the items are old news but I think it is a good checklist that should be on the boiler plate for web application design documents. By putting security requirements in the software specification and design documents, the project manager can then allocate time and resources to security aspects of development. In addition, it reminds developers to ask themselves if the software is meeting those requirements throughout the development process. This is opposed to thinking about security after the entire application has been written and discovering a fundamental design flaw that will require re-writing a good portion of the application.

I particularly appreciate that each item on the CWE/SANS list is weighted including weakness prevalence, remediation cost, attack frequency, attacker awareness, etc. No project has an unlimited budget but you can prioritize on where to focus your resources to achieve the most secure solution. Generally it is a good idea to ensure that the cost of defeating an application’s security far outweighs any benefits to be gained from doing so. The cost of defeating an application might include labor time, computing resources, fines, and threat of jail time with a cell mate named Bubba, etc.

It is quite a challenge to develop secure web applications because generally by their nature they need to accept user input. I believe that it is typically much more difficult develop a secure system than it is to break in to the system given the same number of hours so there is often more burden on the developer. It might take only two or three days to develop a working database driven web application but many additional weeks to harden it against attacks and make it reliable, scalable, and highly available. Including security requirements in the software specification and design is essential to planning and allocating resources.

Ideally automated tests should be included to continuously test vulnerabilities throughout the life of an application. This way security vulnerabilities introduced by code changes will be detected early in the development process instead of later in production. Automated tests could attempt buffer overflows, sql injections, etc. and could be executed prior to a developer’s check-in or on a nightly cron job that automatically checks out the code and runs the tests against it. Although costly to implement initially, automated security testing will likely pay for itself many times over the course of an application’s life. I plan to talk more about automated testing in future posts.

January 13, 2009