In search of a client-side templating solution

After a long time I am posting a technical entry. I am in search of non intrusive client-side templating solution. The following are the requirements

• Non intrusive Should not require something like

ko.observable, ko.mapping and then tableMetadataViewModel.name(), tableMetadataViewModel.name(‘Employees’)

Ember.Object.create, tableMetadataModel.get(‘name’), tableMetadataModel.set(‘name’, ‘Employees’)

The problem with these kind of solutions is that these solutions are no longer Plain Old JavaScript Objects. They introduce a tight coupling with the templating library / framework which is implicit.

• Is reactive / supports ‘Live Templates’ To Quote from Wikipedia

For example, in an imperative programming setting, would mean that is being assigned the result of in the instant the expression is evaluated. Later, the values of and can be changed with no effect on the value of .

In reactive programming, the value of would be automatically updated based on the new values.

A modern spreadsheet program is an example of reactive programming. Spreadsheet cells can contain literal values, or formulas such as "=B1+C1" that are evaluated based on other cells. Whenever the value of the other cells change, the value of the formula is automatically updated.

In this specific context, when I update the name of the table object by setting an alias like

table.alias = ‘e’

It should automatically update in the UI.

• Two-way data-binding

when I update the name of the table object by setting an alias like

table.alias = ‘e’

It should automatically update in the UI.

And when I update the alias in the UI, it should update the object model.

• Model as the single-source source of truth In this specific context, when I update the name of the table object by setting an alias like

table.alias = ‘e’

All the places in the UI where table is rendered as a view should be updated.

Reactivity and Two-way data-binding go a long way in helping Model as the single source of truth.

• Modular

Supports defining templates in external files and plays nice with requirejs text plugin.

• Performs efficiently

Supports r.js optimizer & requirejs text plugin

For example splitting the application into small modules that have a single responsibility requires writing many modular templates. But i may result in poor performance if we make many calls to load them from the browser. Bundling them with the r.js optimizer for example can reduce this performance hit.

Granular re-rendering

Reactive templates should not mean the whole template is re-rendered every time model changes.

For example modifying an item, adding an item or deleting an item in a list should not re-render the whole template and instead just re-render / add / delete the concerned item nodes and the whole list or worse the whole template.

No dirty checking

For example angularjs uses dirty checking that impacts the performance negatively.

Read this to see what I mean http://stackoverflow.com/a/18381836

• Plays nice with other legacy frameworks / libraries

Recently I tried my hand with Polymer Web Components Stack. I ran into a hard to identify issue. Finally the issue was joint.js(+jquery) which I was using in conjunction with polymer was walking the DOM tree and made an assumption that the root node is the document node. Which is not true when it comes to shadow dom.

• Fits well with Nicholas Zakas’s Scalable Application Architecture

Can I find the templating solution that fits well with Nicholas Zakas’s Scalable Application Architecture? Time has to tell.

http://www.youtube.com/watch?v=vXjVFPosQHw

object.observe, web components based?

Some sort of a dom based templating solution based on object.observe on the modern browsers and an object.observe polyfill in other browsers which do not support object.observe should be good. But looks like knockout, ember etc are not taking up the object.observe route yet. May be I will end up forking knockout and implement one myself.

A build-time two-step transform view?

http://martinfowler.com/eaaCatalog/twoStepView.html with one of the transforms / templating running at build-time may help support multiple template engines from a single template. But it will be really hard to build such a library.

Taking a hard dependency on a single templating solution, esp. for a product does not seem right.

Are there any adapters / abstractions available that help one switch templating solutions over the same Model?

Any suggestions comments would be welcome.

Ron Jeffries on Estimation

Ron Jeffries has an excellent write up on estimation on the Pragmatic Programmer website

On Big Bang rewrites

The C3 project’s purpose was to replace the entire family of Chrysler payroll programs. It didn’t accomplish that. Many years later, the subsequent project to replace the entire family of payroll programs has also not accomplished it. We now realize what should have been done.

We should have replaced the broken bits, one at a time, most valuable first.

The fundamental idea of making a complete list of everything payroll, and clicking through it, was wrong.

On (Gu)es(s)timates

It seems that “they” often want to know how long something is going to take, and how much it will cost. My view is that “they” don’t even know what they want, so we bloody well can’t possibly know how long it will take. However, “they” are often powerful and have the money we need, so we need to answer their question, even though we cannot.

Most of the time, “they” know how many people they’ll give us, and how much time. They do that head-shaking thing until we “estimate” the numbers they have in mind. So we should turn the question around. “How much did you want to spend, and when did you hope to see the product?” Then they tell us, and we decide whether we can do something reasonable within that budget. If we can, we go ahead. If not, we politely decline the business.

On vague product ideas

If no one knows, and no one has an idea, odds are the project is too vague and too big. Run away. However, if for some reason you enjoy leaping off cliffs hoping to generate a plan on the way down, here’s something to try. Chet Hendrickson and I call it the “Five Card Method.”

http://pragprog.com/magazines/2013-02/estimation-is-evil

Happy reading.

Server Communication

This post is part of a series of posts on Client-Side Web Application Development using JavaScript.

You need to communicate with the server to retrieve the model from the server, and to put the updates to the model on the server. These libraries typically should provide a higher level of abstraction than xhr (or even $.ajax). Typically they should handle

• The model serialization / de-serialization (ex: date format issues). Most of the community as settled on JSON as the default format here.
• Protocol used to communicate with the server (REST / DDP – See below)
• Provide a higher level of abstraction to deal with asynchronous ajax calls like jQuery’s Deferred / Promise abstraction.
• Change tracking?
• Isolate and abstract the server communication from the rest of the application modules.

There are libraries that provide this kind of functionality out of the box. Ex: Backbone.sync, Knockout-REST

In the Requirement Specific Parameters post I talked about live / real-time updates. Meteor implements real-time updates on top of a protocol called Distributed Data Protocol (DDP). This alleviates us from a need to poll the server for updates. Other frameworks like Derby also support this real-time updates, they call it Subscriptions. I am hoping in the future everyone will settle on a standard protocol.

URL Design, Routing, Navigation and History

This post is part of a series of posts on Client-Side Web Application Development using JavaScript.

Some of my notes from links given below:

Explicit URL Design

URL Design is the process of designing the structure of your application’s URL. Do not leave URL Design to the framework / library you are using. Make an explicit choice about the URLs used by your application.

Routing

In a Single Page Application you want the URL to change as you navigate between views. This will provide the application the ability to deep link into specific parts of the application. Routing is the process of selecting a view based on the URL.

Location Hashes

In a Single Page Application, when the URL changes you do not want to go the server to get the page when the URL changes. This is achieved typically by using location hashes in the URLs. Location hashes are supported in all the browsers, but they are not understood by the server.

When when the Search Engine Bot sends a Hash url to the server the server the server ignores the hash and will not be returning the right page. People workaround this by using a hack called hash bangs that is supported by Google’s Search bot. Hash bangs have been criticized by several Smart people and are not recommended.

HTML5 History API – pushState API

The alternative that is acceptable is HTML5 History – pushState based URLs. This allow to server pages in the client without going to the server on a URL change. But the issue with HTML5 history – pushState API is that it is not supported in all the browsers. So the routing engine / history library you select should be a Polyfill for unsupported browsers.

What ever polyfill hack / workaround the client-side library uses needs to be understood by the Web Server and the right pages should be served for SEO scenarios.

Stateful Routing

Traditionally these routing engine implementations were inspired their counterparts on the server-side. Server-side is a stateless environment so these routing engines on the server were designed with that limitation in mind. The client-side on the other hand is a different story where you can leverage the stateful nature. Ember’s routing library implements a stateful routing. It is something to keep an eye on.

Parameters

Can you deep link to specific views in your Single Page Application?

If you can deep link are the URLs SEO friendly, are they readable?

Should you use location hashes which are supported in pretty much all browsers or should you use HTML5 Push state based navigation bowser support for which is still poor?

Do you need SEO? SEO requirement has an impact on location hashes. Can you avoid Hash bangs if you choose the right routing / navigation / history library?

Reading List

http://warpspire.com/posts/url-design/

http://codebrief.com/2012/03/make-the-most-of-your-routes/

http://blog.benward.me/post/3231388630

http://www.adequatelygood.com/2010/7/Saner-HTML5-History-Management

Template Engines

This post is part of a series of posts on Client-Side Web Application Development using JavaScript.

What is a Template Engine?

JavaScript templates are essentially a way to address the need to populate an HTML view with data in a better way than having to write a big, ugly string concatenation expression.

from – http://www.dehats.com/drupal/?q=node/107

Categories

On an abstract level most template engines implement a Domain-Specific Language (DSL) to marry HTML with your JavaScript objects. Template engines can either work with the DOM or with plain strings. Based on these we can classify the template engines as

• DOM based, DSL based (ex: Angular, Knockout)
• DOM based, Non DSL – (ex: Future browsers that will natively support DOM templating)
• String based, DSL based (ex: Handlebars, Mustache, dustjs)
• String based, Non DSL – (Plain Old Html ex: Plates)

Why is the categorization relevant?

• Using a DSL based template engine is not so user interface designer friendly, these templates are no longer Html.
• Using a DOM based engine makes it difficult to execute the template on the server-side. These factors have a significant impact on template engine selection.

Isomorphic Templates

I have written about some requirement specific parameters here, that are problematic if we use a client-side template engine. For the cases where we need

• SEO / Accessibility / Progressive Enhancements
• Faster initial load times

a workable solution available today is to

• Detect these scenarios
• Render the pages on the server-side rather than on the client-side.

Of course we do not want to duplicate the templates once for the server and once for the client. So for these scenarios we need a template that is isomorphic (can execute both on the server and client).

A string based template engine is the best choice for the scenario, But if you use a DOM based template engine it is still not the end of the road, there are server-side DOM simulations available.

Keep in mind

• These are DOM simulations are not real browser DOMs
• And are bleeding edge

Template engine performance

Template engine performance has a context associated to it. That is where the template is rendered.

On the client

• String based: string => template engine => string => browser => DOM
• DOM based: DOM => template engine => DOM

On the server

• String based: string => template engine => string
• DOM based: string => DOM Simulation => DOM => template engine => string Of these operations string => DOM is an expensive operation. So in theory string based template engines are faster on the server and DOM based template engines are faster on the browser. But its good to actually measure it.

The other point to keep in mind is when considering a string based template engine on the client we also need to take into account the time taken to load the template output into the DOM.

String based: string => template engine => string => browser => DOM

Notes summarized from

https://github.com/leonidas/transparency/wiki/Defining-template-engine-performance

and

http://blog.linvo.org/post/22962778793/microtemplating-vs-dom-welding

Updates / Refresh

Another feature that you may want to consider when you evaluate a template engine is whether the template engine can update the views automatically when the (view )model changes. Can the template engine re-render when the model changes and can that be localized to partial portions of the view or the entire view has to be refreshed. Different libraries use different terminology here, Spark – Meteor uses the term Live Page Update, Ember calls this Auto updating templates.

Recommendations

• Use a DOM-based template engine if SEO / Accessibility readers / Progressive enhancement are not a concern & you do not need server-side rendering. Ex: Knockout
• When both client & server-side rendering are required choose a string based template engine. Ex: Handlebars / dustjs.
• Even if you use a string based template engine it is worthwhile to keep a watch on the following categories. DOM based template engines (Angular, Knockout). Non DSL based template engine that lets you write plain old html markup and uses standard json (Plates).

Logic in templates & Reuse

• If you come from a server-side scripting environment you may be familiar with logic getting embedded in the templates. Logic-less templates are an answer to this. DOM-based template engines are generally not Turing-Complete Programming Languages so it is hard to embed logic in DOM-based templates
• If you come from ASP.NET Web Forms side you may be familiar with the Controls model which offers template reuse at control / widget level. Angular has support for something like this, they call it as directives.

Reading List

http://www.dehats.com/drupal/?q=node/107

http://blog.linvo.org/post/22962778793/microtemplating-vs-dom-welding

https://github.com/leonidas/transparency/wiki/Defining-template-engine-performance

http://engineering.linkedin.com/frontend/client-side-templating-throwdown-mustache-handlebars-dustjs-and-more

Data-binding

This post is part of a series of posts on Client-Side Web Application Development using JavaScript.

The marriage of Html and your (view )model data does not end with template engines alone, because it is just one side of the story. A traditional template engine is usually about getting your data on to the browser as Html, what about the other way round. When the user inputs a value in the browser can the template engine update your model back. This called bi-directional data-binding. Some people use the term data-linking.

Many of the DOM based template engines support this bi-directional data-binding out of the box. Some of the string based template engines also support this (Ember for example implements this feature on top of handlebars).

This feature is particularly useful if your application is input intensive. Read more about Input intensive applications here.

Structuring presentation logic using UI Architectural patterns

This post is part of a series of posts on Client-Side Web Application Development using JavaScript.

Classics

No discussion about UI Architectural patterns is complete without reading / linking to Martin Fowler’s GUI Architectures & Michael Feather’s The Humble Dialog Box. I will quote extensively from Fowler’s work here, to drive point. If you know MVC & Presentation Model (or MVVM) really well, You can jump directly to the MVC vs. MVVM section

Separation of Concerns

These UI Architectural patterns are all about Separation of Concerns and having more maintainable code – Divide and Conquer

Model View Controller (MVC)

• In MVC, the domain element is referred to as the model. Model objects are completely ignorant of the UI.
• Make a strong separation between presentation (view & controller) and domain (model) –Separated Presentation.
• Divide GUI widgets into a controller (for reacting to user stimulus) and view (for displaying the state of the model). Controller and view should (mostly) not communicate directly but through the model.
• Have views (and controllers) observe the model to allow multiple widgets to update without needed to communicate directly – Observer Synchronization.

Examples of Javascript Frameworks / libraries which help you with structuring your application based on MVC pattern- Backbone Core, Ember.js, Spine, CanJS

Model-View-View Model

(Presentation Model / MVVM)

The essence of a Presentation Model is of a fully self-contained class that represents all the data and behavior of the UI window, but without any of the controls used to render that UI on the screen. A view then simply projects the state of the presentation model onto the glass.

To do this the Presentation Model will have data fields for all the dynamic information of the view. This won’t just include the contents of controls, but also things like whether or not they are enabled. In general the Presentation Model does not need to hold all of this control state (which would be lot) but any state that may change during the interaction of the user. So if a field is always enabled, there won’t be extra data for its state in the Presentation Model.

Since the Presentation Model contains data that the view needs to display the controls you need to synchronize the Presentation Model with the view. This synchronization usually needs to be tighter than synchronization with the domain – screen synchronization is not sufficient, you’ll need field or key synchronization.

Presentation Model is not a GUI friendly facade to a specific domain object. Instead it is easier to consider Presentation Model as an abstract of the view that is not dependent on a specific GUI framework.

 

Examples of JavaScript Frameworks / libraries which help you with structuring your application based on MVC pattern – Knockout, Knockback, Kendo UI MVVM (Commercial).

Not all the JavaScript frameworks implement MVC in the strict small talk MVC sense, not even they implement it in the struts sense. They improvise it for the JavaScript scenario. Most of them agree on Separated Presentation and have Model & View. Some of them do not have controllers. Some call controllers as routers. Some implement controller functionality in views. In general all these are variations over MVC – MV*. Model View Whatever – MVW is another term that you would see being thrown around. [from Journey through the JavaScript MVC Jungle]

MVC vs. MVVM

MVC

• MVC is about separating business logic (Model) from the presentation logic (View and Controller) – Separation of Concerns
• It opens up the possibility of having multiple presentations technologies on the same model.

MVVM / Presentation Model

While MVC addressed separation of Business Concerns from that of the Presentation, It did not focus much on Presentation Logic. The Presentation Logic in modern applications is significantly complex. Presentation Model puts additional emphasis on organizing presentation logic (which was just a high level guidance in original MVC – Controllers handle user input and View handle display & display logic. In much of the modern UI Toolkit implementations User Input is sucked into the toolkit itself).

The move to Presentation Model came when the presentation logic in the views (and controllers) became more complex and testability of that  logic became a primary concern. Read the Humble Dialog Box article if you haven’t read it. So Presentation Model is all about moving the presentation logic from the View to the View Models (not as is but in an abstract way that is UI framework independent and becomes unit testable) so that the View becomes really dumb / humble.

But it does not come free, along with moving the Presentation Logic from the view we also moved the Presentation State from the View and the View now needs to synchronize with that. This synchronization can be a pain in environments that do not support data-binding.

• MVVM is about separating business logic (Model) from the presentation logic (View and View Model) – Separation of Concerns
• But it also addresses other concerns,
  • Having near zero presentation logic & state in the views by moving them to view models
  • Having a view model that is a presentation technology agnostic – UI platform-independent abstraction of a View – Josh Smith’s MSDN Article on MVVM
  • Having testable presentation logic in view models

  MVVM makes much more sense if

• You are towards near zero presentation logic
• You are towards Unit Testable presentation logic
• You are towards two-way data-binding and have a presentation technology that supports it

Some people hate data-binding and its natural for them to use MVC.

Recommendations:

If you’re writing an application that will likely only be communicating with an API or back-end data service, where much of the heavy lifting for viewing or manipulating that data will be occurring in the browser, you may find a JavaScript MV* framework useful

If, however, you’re building an application that still relies on the server for most of the heavy-lifting of Views/pages and you’re just using a little JavaScript or jQuery to make things a little more interactive, an MV framework may be overkill. 

from http://coding.smashingmagazine.com/2012/07/27/journey-through-the-javascript-mvc-jungle/

Use MVVM if possible. If you are not comfortable with it use MVC for now but keep an eye on JavaScript MVVM frameworks, they may become a standard in the future.

• I am biased towards MVVM and all for two way data binding (I also happen to have worked on Windows Presentation Foundation – WPF). So it is a personal preference.
• The next version of EcmaScript (the standard for JavaScript) code named Harmony (ES-Harmony) will have better support for observable properties (VB / .NET style properties with getter and setter) and more future browsers will support HTML5 data binding attributes (data-* attributes). Both of these future directions can make data binding easier with html5 & JavaScript.

Reading List

GUI Architectures

The Humble Dialog Box.

Presentation Model

http://coding.smashingmagazine.com/2012/07/27/journey-through-the-javascript-mvc-jungle/

http://addyosmani.com/largescalejavascript/

Modular JavaScript

This post is part of a series of posts on Client-Side Web Application Development using JavaScript.

Modularity

I read this quote somewhere, I cannot recall, But it best echoes the sentiments about modular applications.

The best way to write a large-scale application is by not writing one, instead choose to implement small rightly coupled modules

The goals

• Do not pollute the global namespace
  • Avoid namespace collisions
• Load modules fast (even out of order, given the asynchronous nature of the browser)
• Execute / Evaluate modules in the right order

Optimizers

Script Loaders load scripts asynchronously and module libraries handle dependencies, that does not mean that we leave performance out of the equation and load 20 JavaScript files asynchronously with round-trips to the server. Nor does it mean to dump all the JavaScript required for the page in one single file. Most of the module libraries use a optimization tool that can bundle all the scripts into a single file for deployment. Example requirejs used r.js and curl.js uses cram. It may be good idea still to leave it to the Script Loader implementation of the Module Library when the scripts are conditionally required (for ex: a Canvas library to mimic HTML5 canvas in a older browser).

Standards

So we decided to implement modular JavaScript in our application, rather inventing a way for ourselves to implement modules in JavaScript we can choose to embrace one of the following standards.

• Asynchronous Module Definition format (AMD)
• CommonJS (CJS) module format
• Node (though this is not a standard you can consider this as a slightly modified version of CJS. I hope CJS adopts the additional features from Node)

Recommendation

Which one should my application use, I will use the following recommendation (slightly modified for node) from John Hann’s blog post.

1. write modules that could execute in a server environment in CJS
2. write modules that could benefit from AMD’s browser-friendly features in AMD format

What if you need a module to executed in both client and server? Use requirejs to run AMD in node instead of using Node’s inbuilt module loader and module format? The solutions like onejs / browserify seem like too much magic to me.

Reading list

http://requirejs.org/docs/whyamd.html

http://unscriptable.com/2011/09/30/amd-versus-cjs-whats-the-best-format/

http://blog.millermedeiros.com/amd-is-better-for-the-web-than-commonjs-modules/

http://briancavalier.com/presentations/pgh-js-amd-10-2011/#0

http://addyosmani.com/writing-modular-js/

http://msdn.microsoft.com/en-us/magazine/hh227261.aspx

From mud to structure

This post is part of a series of posts on Client-Side Web Application Development using JavaScript.

Unless you put conscious effort in structuring your system, It ends up as a “Big Ball of Mud”. With JavaScript it is easier to get into this because

• Unless you take special care everything ends up as Globals
• JavaScript does not support Namespaces out of the box (consider this as logical organization of JavaScript code)
• JavaScript does not support the concept of modules specifying dependencies between them out of the box (something like .NET Assemblies, AssemblyRefs in assembly metadata).

Modularity

Even though it is not possible to do these natively in the language using language specific constructs / support from the virtual machine, JavaScript being a really flexible language (it will be because it was inspired by LISP), it is possible to mimic those in multiple ways. Rather than inventing our own way to do these tasks there standard ways to implement modular JavaScript solutions.

UI Architectural Patterns

The other thing about User Interface logic is that it is the most difficult to organize. Architectural patterns can help here.  Separated Presentation patterns help you separate presentation logic from the business logic (ex: MVC) and provide a way to add structure to your application

Summary

So it is recommended that you use write

• Modular JavaScript using a library that implements a Standard Module pattern.
• Use a MV* framework that lets you implement one of the popular UI Architectural patterns

otherwise you application would end up as a “big ball of mud”.

A bare minimum JavaScript UI Stack

This post is part of a series of posts on Client-Side Web Application Development using JavaScript.

A bare minimum JavaScript UI Stack should have libraries / frameworks handling these concerns.

• A module / script loader library – to structure code into modules, handle dependencies between them
• A server communication library to retrieve the model from server and update it back.
• A MV* library / framework – to structure code as per one of the popular UI architectural patterns
• A Routing library to select a View Template based on the URL
• A Template engine to render dynamic Views from the model data
• A data-binding library to update data from the model – view and vice versa.
• A UI widget library to provide commonly used UI Widgets

Of course I have made a lot of assumptions about the web application here, but you get the idea.