Nodejs Production Strategy : Async, Logging and Event loops

Standard

If you are or planning to run Nodejs (Express) in production for your solution then you should avoid certain mistakes that most developers do when deploying the application in production that may include using the – Not so useful logging packages, Not checking if the application is having a event loop block etc.

I have tried to explain the concept and suggest some solutions that might help you find your own solutions in longer run.

“Node.js application servers are single thread deployments & Parallelism does not exist from the programmer’s perspective as its I/O bound rather then CPU bound.”

 

(1) Writing Async Workflows

That means if you wrote code not following async patterns or workflow then certain loops can make the application slow. Let me explain this with example, we can assume you have a web application that has few thousand active users and all have a common feature of authenticating user before accessing core features. If you didn’t write the code using async workflows then other users would have to queue and that can be painful. Here is a list of packages that can help you write async workflows or pipelines.

(2) Unblock Event Loops

As mentioned the node.js process is a single thread deployment meaning that anything that blocks the event loop would block almost everything. Here is a good post on understanding event loop.

After you understand what is a event loop then you can easily see that blocking can be serious issue. Let me try to explain the same with a use case, lets assume you have a event where you have to parse large files as user uploads. If you didn’t write async functions to handle it would freeze the application server and you might not even have a single clue what just happened. To debug this a fantastic tool exists named “blocked“. The logic to this package is pretty simple it calculates time difference by getting the time at two specific time instances.

To use in your express app :


var blocked = require('blocked');

//Function01 set after 500ms
setInterval(function(){
//Some heavy computing
}, 500);

//Function01 set after 3000ms
setInterval(function(){
//Some heavy computing
}, 3000); //Difference between two and tell if it was blocked.
blocked(function(ms){
console.log('BLOCKED FOR %sms', ms | 0);
});

Other way is using procmon that gives a UI for seeing event loop delays etc

(3) JSON Logging
It might b a good idea to log everything in JSON as its easy to query and off course you can bind events in case of error handling. Using something better then console.log() is good idea because :

  • Built in serializers for errors, response and objects.
  • Where logs are saved and processed.
  • Adding more parameters in the log format including host-name, process id and application name. I would recommend binding API paths to their role of function as helps detect what happened when.
  • Log file rotation and more can be added.

In order to do that, check node-bunyan. It is straight forward as it allows you to create a logger variable and then use it with logging mechanism. The output is all JSON which is awesome.

More production aspects coming soon!

 

Share Button

ngCordova : Device APIs for your Hybrid mobile apps.

Standard

As I discussed in my last post that discussed about how ionic apps can be a lifesaver for javascript developers producing production mobile apps without learning multitude of different programming languages. I stick to Javascript for good here :-)

The aspect that one needs to learn while developing a hybrid mobile solution  (other than javascript) is Cordova plugins. As you all know that every mobile app needs to be efficient and smart enough to utilise the device platform that can be essential for any mobile strategy may it be customer engagement (using Push notifications) or simply uploading pictures to the system (using camera for picture click and upload) etc.

Offcourse Cordova is a open style development repository that can be challenging if you don’t have any device API experience. A solution to this problem can be exploring ngCordova which is a a collection of AngularJS wrappers done right for ionic framework. It also has the right documentation to get started and rock in production setups. 

I would be focusing on IOS platform for now and its worth mentioning TouchID is supported here and that is just awesome. Let me walk you through the process to setup and add a single plugin below.

  • Installing ngCordova to your Ionic app : You need to add it in your dependencies in “bower.json” and install it. The other way is to install it directly 
$bower install ngcordova
  • Goto your “www/index.html” and add : 

<script src=”lib/ngCordova/dist/ng-cordova.js”></script>

<script src=”cordova.js”></script>

  • Inject in Angular Module 
angular.module('myApp', ['ngCordova'])
  • Avoid fully loaded device so add in your ngcordova.js before using any plugin

$ionicPlatform.ready(function() {$cordovaPlugin.someFunction().then(success, error);});

Now how to add a Cordova plugin such as TouchID (offcourse you can add any plugin using the same process).

  • STEP 1: Create a Ionic project and test it works

$ionic start test tabs

$cd test

$ionic platform add ios

$ionic serve

  • STEP 2: Add touchID plugin 

$cordova plugin add cordova-plugin-touchid

  • STEP 3: Add “ng-cordova.js” and inject in angular dependency as mentioned above. 
  • STEP 4 : Writing testController that would call for authentication, check if successful and if not then pass error and offcourse log the error in console.

.controller("testController", function($scope, $ionicPlatform, $cordovaTouchID) {

$ionicPlatform.ready(function() {

$cordovaTouchID.checkSupport().then(function() {

$cordovaTouchID.authenticate("You must authenticate").then(function() { alert("The authentication was successful"); },

function(error) {console.log(JSON.stringify(error)); }

);

}, function(error) { console.log(JSON.stringify(error)); });

});

})

Happy coding!

Share Button

Building rockstar Hybrid mobile apps using Javascript

Standard

You are a “WEB-ONLY” nodejs developer and realise that most of the productive businesses are trending to go mobile only (sooner or later). Ok, maybe you build responsive apps that somehow blend to work with browsers but web standards cannot directly use the mobile API’s. This means powerful products cannot utilise the interaction with users in realtime using mobile api’s such as push notification, access etc.

Well a few months  back I would have asked you to goto “Apache Cordova” which is a rock solid platform to build hybrid apps but it sure misses the creative part i.e. somehow easy integration with node frameworks such as express and a clientside MVC attachment.

Hybrid apps have HTML,CSS and Javascript as the ruling code where as they can work to behave same as native apps.

Ionic is answer! It packs a Angular as its client side MVC, easily integration with Express framework and few awesome theme options such as IONIC-MATERIAL. Offcourse bringing express to the table adds a possibility to open a lot of opportunities that can help easily integrate with present node app or new architecture written in node. On top of all this it has a code generator for you to choose from start i.e. “blank”, “sidemenu” and “tabs”. Lastly it supports both IOS and Android production that you can test and launch in various app stores. 

To get started, you can install node on your local system and follow the steps :
INSTALL : $npm install -g cordova ionic
CREATE APP : $ionic start app_name sidemenu/tabs/*nothing*
$cd app_name
DEVELOP :$ionic serve (this would start the app on localhost:8100).
ADD PLATFORM :$ionic add platform ios/android
BUILD IT : $ionic emulate ios/android

For setting up emulators, visit here for IOS and here for Android

Add material design to your ionic app, check here.

Happy hacking!

Share Button

Why I choose to code in Javascript?

Standard

Javascript was created by Netscape as Livescript. It always has been misunderstood over its potential as its a “Scripting” Language then programming language. Although, it has been there for many years and has gone through many transitions over execution and implementation improvements.

It offers solutions to many problems that I have tried to sum and share for you.

They are as follows:

  • Solution 1 : Unify Client side and Server side code 

There have been many attempts to do this in the past but it has always been un accepted by the developer community and most of them failed miserably bad. Although this is important aspect that is required to power scalability and acceptance, sync among developer teams. Off-course code wise it is important as well because one can reuse various components and resources.

Tip : Angular/Backbone and NodeJS/Express all work on same javascript principles yet serve the client and server side duties when deployed.

  • Solution 2: Promoting the Non-Blocking programming 

Javascript on stack follows the async architecture – this is easy to understand if you have a clear understanding of AJAX. In Async, lets say two events (One complex, One simple) have to execute, then the output is generated first for simple event as the complex one will take time to execute i.e not halting the process. This makes the system extremely fast. Another aspect is inbuilt Callbacks and Event loops in the language. Ok callbacks are not that tough to do in most other languages but here it is build-in binding it to the I/O event loop making it easy for developers to write event driven functions that do callbacks as part of inbuilt functions.

Let me explain this with example :

Consider Both the codes that will give you same outputs but the difference is :  First one follows sync architecture whereas other one follows async architecture.


function getUser(id) {
var user = db.query(id);
return user;
}

console.log('Name: ' + getUser(432).name);

The function blocks until the database call is completed. This means the server is doing nothing but waiting until the function completes, ignoring other pending requests.


function getUser(id, callback) {
db.query(id, callback);
}

getUser(432, function (user) {
console.log(user.name);
});

console.log('Done');

The function will first show “Done” and then show the username that is fetched from the database.

A much better approach of Non-blocking programming can be seen in this code where callback is used :

function add(list, title, position, callback) {
// Add new item to 'items' store
db.insert('items', { list: list, title: title }, function (item) {
// Set position if requested
if (position !== null) {
db.get('items.sort', list, function (sort) {
addToListAt(sort, item.id, position);
db.update('items.sort', sort, function () {
finish(item.id);
});
});
}
else {
finish(item.id);
}
});
function finish(id) {
// Perform final processing
callback({ id: id, status: 'ok', time: (new Date()).getTime() });
}
}

The if else simplifies the function. The skill to learn here is that one needs to think in this approach if you have write good javascript code i.e think about functions as event loops. This style is a bit tricky to catchup and even a bit more tricky while debugging.

  • Solution 3 : Fast prototyping 

Object-Oriented Programming without classes (and without endless hierarchies of classes) allows for fast development (create objects, add methods, and use them). This reduces refactoring time during maintenance tasks by allowing the programmer to modify instances of objects instead of classes. This speed and flexibility paves the way for rapid development.

  • Solution 4: Functional Programming

Its easy to create a library of reusing functions that can play important role in multiple components. Its very easy to integrate this to new projects. In functional programming the function depends upon the arguments not the inputs.

 

 

 

 

Share Button

Sharing simple file upload snippet in Javascript

file-uploads1
Standard

File Upload is important part of any web application. May it be a simple image album or a complex enterprise system, its required everywhere.

Facts :Since the application revolves around Restful, lets get the verbs clear :

  • Node.js doesn’t allows you to peep into directories and get download the files where these directories are generally private unless you make it as Public Folder.
  • In any NoSQL Store, you can create many collections. Lets say every collection corresponds to a specific collection.
  • When http://example.com/resource
  1. GET : List of Members of the collection, complete with their member URIs for further navigation.
  2. PUT : Replace entire collection with another collection.
  3. POST: Create a new entry in the collection where the ID is assigned by the collection.
  4. DELETE: Delete the entire collection.
  • When http://example.com/resource/IDVALUE
  1. GET : Retrieve a representation of the addressed member of the collection expressed in an appropriate MIME type.
  2. PUT : Update the addressed member of the collection or create it with the specified ID.
  3. POST: Treats the addressed member as a collection in its own right and creates a new subordinate of it.
  4. DELETE: Delete the addressed member of the collection.
  • In NodeJS, If using express for earlier version then you can use bodyParser() middleware. Although, If you are using the latest express version then that specific middleware is removed.
  • In NodJS, the file information is saved in req.files.

The logic :

  • STEP 1: The index.html

<!DOCTYPE html>
<html lang=”en” ng-app=”APP”>
<head>
<meta charset=”UTF-8″>
<title>Simple FileUpload Module</title>
</head>

<body>
<form method=’post’ action=’upload’ enctype=”multipart/form-data”>
<input type='(write F_I_L_E here)’ name=’fileUploaded’>
<input type=’submit’>
</body>
</html>

  • STEP 2 : Defining /Upload with app.post and also showing the uploading confirmation on terminal

var express = require('express'); //Express Web Server
var busboy = require('connect-busboy'); //middleware for form/file upload
var path = require('path'); //used for file path
var fs = require('fs-extra'); //File System - for file manipulation

var app = express();
app.use(busboy());
app.use(express.static(path.join(__dirname, 'public')));

app.route('/upload')
.post(function (req, res, next) {

//Function to upload the file and pass it to the folder

var fstream;
req.pipe(req.busboy);
req.busboy.on('file', function (fieldname, file, filename) {
console.log("Uploading: " + filename);

//Path where image will be uploaded
fstream = fs.createWriteStream(__dirname + '/public/file/' + filename);
file.pipe(fstream);
fstream.on('close', function () {
console.log("Upload Finished of " + filename);
res.redirect('back'); //where to go next
});
});
});

var server = app.listen(3030, function() {
console.log('Listening on port %d', server.address().port);
});

You can download the working code here on Github : https://github.com/ga1989/fileupload-nodejs

Share Button