Simon Baynes : Software Architect

Google AMP, My Take

Simon Baynes — Thu, 07 Apr 2016 16:26:58 GMT

Personally I think Google AMP is a travesty. It is pitched as a way to speed up content delivery on mobile, which it undoubtedly achieves. However, it does this by bullying you into having to do it, as you will be penalised in search results for not having an AMP version of your content. It is not Google's role to police the Worldwide Web. Stopping publishers from jamming their pages with ads, etc and slowing down their pages. Improving this situation should be driven by market forces, and not what Google thinks. What is wrong with responsive design? Google AMP feels like the bad old days of building WML pages.

Personally I think Google AMP is a travesty.

Google AMP is limited, it does not allow you to do lots of things that a modern publisher would like to be able to do. A good example of this is real-time updates. There is nothing in the current standard that will support live updates in an article, there is just an issue on GitHub which has some underwhelming suggestions. If you want to do this in standard HTML, no problem there are several options as long as you can scale your infrastructure. This is one of my main issues, that of no competitive advantage. If you have a great idea you want to deliver and you want to put it in front of your growing mobile audience then if AMP doesn't support it then you are going to have to make suggestions to the specification, and wait and see. This is terrible. AMP by its nature will always trail behind what can be done with HTML and JavaScript.

AMP adds another presentation you have to maintain. AMP doesn't replace the mobile representation of you pages, it adds another one. This means that a customer coming from a link in an email will end up on you standard HTML pages, whereas one coming from a Google search will see the AMP version. This adds a serious additional expense to publishers in design, development and testing. It also creates a confusing experience for customers.

AMP adds another presentation you have to maintain.

So coming back to my original point that this was pitched as a way to speed up content delivery on mobile. I actually do not believe Google at all on this. For me it is something they had to do in response to the advertising dollar land grabs made by Facebook with Instant Article, and Apple with Apple News. So my cynical view is that this is just a defensive manoeuvre by Google to protect its revenues.

So my cynical view is that this is just a defensive manoeuvre by Google to protect its revenues.

Introducing Html2Markdown

Simon Baynes — Wed, 02 Dec 2015 16:06:02 GMT

I originally created Html2Markdown in 2013, as a .Net class library. So I could create Markdown versions of my blog posts, so I could migrate to a new blogging engine. Back then, I only needed support for a subset of the standard, so that was all I included.

About six months later I received my first pull request adding

support. It was at this point it seemed other people had use cases, so I decided to finish support for the Markdown specification.

Try it out

If you want to try it out I've created an application that showcases what it can do.

http://html2markdown.bayn.es

NuGet

If you want to include this library in your project then it is available as a NuGet package.
PM> Install-Package Html2Markdown

Real-time Web Apps with Server-Sent Events (pt 2)

Simon Baynes — Thu, 20 Aug 2015 05:27:18 GMT

This is the second part of a two part series about building real-time web applications with server-sent events.

Building Web Apps with Server-Sent Events - Part 1

Reconnecting

In this post we are going to look at handling reconnection if the browser loses contact with the server. Thankfully the native JavaScript functionality for SSEs (the EventSource) handles this natively. You just need to make sure that your server-side implementation supports the mechanism.

When the server reconnects your SSE end point it will send a special HTTP header Last-Event-Id in the reconnection request. In the previous part of this blog series we looked at just sending events with the data component. Which looked something like this:-

data: The payload we are sending\n\n

Now while this is enough to make the events make it to your client-side implementation. We need more information to handle reconnection. To do this we need to add an event id to the output.

E.g.

id: 1439887379635\n
data: The payload we are sending\n\n

The important thing to understand here is that each event needs a unique identifier, so that the client can communicate back to the server (using the Last-Event-Id header) which was the last event it received on reconnection.

Persistence

In our previous example we used Redis Pub/Sub to inform Node.js that it needs to push a new SSE to the client. Redis Pub/Sub is a topic communication which means it will be delivered to all connected clients, and then it will be removed from the topic. So there is no persistence for when clients reconnect. To implement this we need to add a persistence layer and so in this demo I have chosen to use MongoDB.

Essentially we will be pushing events into both Redis and MongoDB. Redis will still be our method of initiating an SSE getting sent to the browser, but we will also be be storing that event into MongoDB so we can query it on a reconnection to get anything we've missed.

The Code

OK so let us look at how we can actually implement this.

Update ServerEvent

We need to update the ServerEvent object to support having an id for an event.

function ServerEvent(name) {
    this.name = name || "";
    this.data = "";
};

ServerEvent.prototype.addData = function(data) {
    var lines = data.split(/\n/);

    for (var i = 0; i < lines.length; i++) {
        var element = lines[i];
        this.data += "data:" + element + "\n";
    }
}

ServerEvent.prototype.payload = function() {
    var payload = "";
    if (this.name != "") {
        payload += "id: " + this.name + "\n";
    }

    payload += this.data;
    return payload + "\n";
}

This is pretty straightforward string manipulation and won't impress anyone, but it is foundation for what will follow.

Store Events in MongoDB

We need to update the post.js code to also store new events in MongoDB.

app.put("/api/post-update", function(req, res) {
    var json = req.body;
    json.timestamp = Date.now();

    eventStorage.save(json).then(function(doc) {
        dataChannel.publish(JSON.stringify(json));
    }, errorHandling);

    res.status(204).end();
});

The event-storage module looks as follows:

var Q = require("q"),
    config = require("./config"),
    mongo = require("mongojs"),
    db = mongo(config.mongoDatabase),
    collection = db.collection(config.mongoScoresCollection);

module.exports.save = function(data) {
    var deferred = Q.defer();
    collection.save(data, function(err, doc){
        if(err) {
            deferred.reject(err);
        }
        else {
            deferred.resolve(doc);
        }
    });

    return deferred.promise;
};

Here we are just using basic MongoDB commands to save a new event into the collection. Yep that is it, we are now additionally persisting the events so they can be retrieved later.

Retrieving Events on Reconnection

When an EventSource reconnects after a disconnection it passes a special header Last-Event-Id. So we need to look for that and return the events that got broadcast while the client was disconnected.

app.get("/api/updates", function(req, res){
    initialiseSSE(req, res);

    if (typeof(req.headers["last-event-id"]) != "undefined") {
        replaySSEs(req, res);
    }
});

function replaySSEs(req, res) {
    var lastId = req.headers["last-event-id"];

    eventStorage.findEventsSince(lastId).then(function(docs) {
        for (var index = 0; index < docs.length; index++) {
            var doc = docs[index];
            var messageEvent = new ServerEvent(doc.timestamp);
            messageEvent.addData(doc.update);
            outputSSE(req, res, messageEvent.payload());
        }
    }, errorHandling);
};

What we are doing here is querying MongoDB for the events that were missed. We then iterate over them and output them to the browser.

The code for querying MongoDB is as follows:

module.exports.findEventsSince = function(lastEventId) {
    var deferred = Q.defer();

    collection.find({
        timestamp: {$gt: Number(lastEventId)}
    })
    .sort({timestamp: 1}, function(err, docs) {
        if (err) {
            deferred.reject(err);
        }
        else {
            deferred.resolve(docs);
        }
    });

    return deferred.promise;
};

Testing

To test this you will need to run both apps at the same time.

node app.js

and

node post.js

Once they are running open two browser windows http://localhost:8181/ and http://localhost:8082/api/post-update

Now you can post updates as before. If you stop app.js but continue posting events, when you restart app.js within 10 seconds the EventSource will reconnect. This will deliver all missed events.

Conclusion

This very simple code gives you a very elegant and powerful push architecture to create real-time apps.

Improvements

A possible improvement would be to render the events from MongoDB server-side when the page is first output. Then we would get updates client-side as they are pushed to the browser.

Download

If you want to play with this application you can fork or browse it on GitHub.

Running a Script on Start Up on Windows 10

Simon Baynes — Mon, 10 Aug 2015 20:53:04 GMT

Previously on Windows 7 I have used the Startup folder technique to run scripts periodically.

I spent some serious time head scratching trying to replicate that on Windows 10. Thankfully I have managed to work it out, with the help of some friends.

Open the run dialog (Windows+R) then type shell:startup then press Enter.

This will open up the startup folder. Once there you can create any shortcut you need.

Happy scripting.

Real-time Web Apps with Server-Sent Events (pt 1)

Simon Baynes — Sun, 09 Aug 2015 07:41:13 GMT

Recently I've been researching how to build real-time web applications, where content is pushed to clients rather than them having to poll or refresh the browser. A lot of people at this point jump straight to websockets. However, these can often be more powerful than you need. Websockets provide a rich protocol to perform bi-directional, full-duplex communication. These are great when you want to do multicast communication between many-to-many clients. If all you need to achieve is one-to-many multicast from the server then Server-Sent Events (SSEs) are a powerful and simpler alternative. SSEs are sent over traditional HTTP; which means you can use a standard webserver rather than getting a websockets server.

Browser Support

At first glance of the CanIUse website it would appear that websockets have better browser support than SSEs. However, there are many polyfills to enable SSEs to function in unsupported browsers.

Example Implementation

Pre-Requisites

The following technologies are required for this demo application.

Build Interface

index.html




    Realtime Demo
        


        Realtime Demo

The main things to focus on here are the creation of an EventSource object.

var source = new EventSource("http://localhost:8081/api/updates");

This creates an open connection to the updates URI. This will be the end point that will be serving our SSEs. Which we can now consume by attaching a handler to the message event.

Now we need to write our Node end point to publish messages.

Server-Sent Events Publishing

app.js

var express = require("express"),
    mustacheExpress = require("mustache-express"),
    dataChannel = require("./custom_modules/data-channel"),
    bodyParser = require("body-parser"),
    app = express();

app.engine('html', mustacheExpress());
app.set('views', './views')
app.set('view engine', 'html');
app.use(express.static("./static"));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({extended: true}));

app.get("/api/updates", function(req, res){
        initialiseSSE(req, res);
});

app.get("/api/post-update", function(req, res) {
    res.render("postupdate", {});
});

app.put("/api/post-update", function(req, res) {
    var json = JSON.stringify(req.body);
    dataChannel.publish(json);
    res.status(204).end();
});

function initialiseSSE(req, res) {
    dataChannel.subscribe(function(channel, message){
        var messageEvent = new ServerEvent();
        messageEvent.addData(message);
        outputSSE(req, res, messageEvent.payload());
    });

    res.set({
        "Content-Type": "text/event-stream",
        "Cache-Control": "no-cache",
        "Connection": "keep-alive",
        "Access-Control-Allow-Origin": "*"
    });

    res.write("retry: 10000\n\n");
}

function outputSSE(req, res, data) {
    res.write(data);
}

function ServerEvent() {
     this.data = "";
};

ServerEvent.prototype.addData = function(data) {
    var lines = data.split(/\n/);

    for (var i = 0; i < lines.length; i++) {
        var element = lines[i];
        this.data += "data:" + element + "\n";
    }
}

ServerEvent.prototype.payload = function() {
    var payload = "";

    payload += this.data;
    return payload + "\n";
}

var server = app.listen(8081, function() {

});

Key parts of this to look at are the headers required to make the SSEs work correctly.

res.set({
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    "Connection": "keep-alive",
    "Access-Control-Allow-Origin": "*"
});

The first three headers are mandatory, but the Access-Control-Allow-Origin is optional, and is how you can control cross domain access with CORS.

Next is the construction of the SSEs. Colin Ihrig does a fine write up of the server side of server-sent events, which I used as a resource to put this together.

data-channel.js

var redis = require("redis");

module.exports.subscribe = function(callback) {
    var subscriber = redis.createClient();

    subscriber.subscribe("liveupdates");

    subscriber.on("error", function(err){
        console.log("Redis error: " + err);
    });

    subscriber.on("message", callback);
};

module.exports.publish = function(data) {
    var publisher = redis.createClient();

    publisher.publish("liveupdates", data);
};

Here we are just utilising the Pub/Sub functionality of Redis, which is really simple as you can see.

To see this working just download the full working application from GitHub. Then browse to http://localhost:8082/api/post-update/ and fill in the form while browsing http://localhost:8081 and you will see the events updating in realtime.

Conclusion

So as you can hopefully see this is really pretty straightforward. Node + Redis hugely simplifies the server-side functionality, and the clientside integration is uncomplicated.

I'll be doing a follow up post on how to handle the reconnection.

New Role

Simon Baynes — Thu, 21 Jan 2010 18:27:40 GMT

Well I am pleased to announce that I have accepted the role of Technical Architect at HostelBookers.com. I had initally set my sights on contracting, but I felt this was an opportunity I would regret turning down.

I am going to really miss my old colleagues at Haymarket. Easily the most talented and hard working people I have ever worked with. I'll be leaving many friends behind which will be a wrench, but thankfully I'm only 20 minutes down the Piccadilly Line so I am sure I'll see them all regularly.

I start on 8th February so I get a nice week of R&R before I start.

Moving On

Simon Baynes — Tue, 05 Jan 2010 22:48:03 GMT

Well after four and a half years I have decided to move on from Haymarket. It was a difficult decision but I felt that I needed a change. I will be leaving behind some proud achievements, great friends and colleagues. I'll be working till the end of the month and then I'll be going contracting, well at least that is the plan anyway.

Wish me luck.

Moving from Apache to IIS

Simon Baynes — Tue, 24 Feb 2009 19:36:07 GMT

My most recent project has been moving all our production websites to IIS from Apache. This mainly quite low level and not at all complicated. The real challenge comes because we are heavy users of mod_rewite.

Fortunately Helicon have a product called ISAPI rewrite 3 which has very similar syntax to modrewrite. There are a few gotchas to bear in mind. The one that I spent too long solving was that the URL that you are assessing with RewriteRule in ISAPI rewrite starts with a / which is not the case in modrewrite. However, I was very thankful to discover that I did not need to change all my rules, I just had to add RewriteBase / just after the RewriteEngine on directive.

The other huge one for me was that environment variables were not supported in ISAPI Rewrite. This did cause me to have to rearchitect thew rewrites I had for my search. However the upside of this was that I got a lot better at rewrites and realised that my initial implementaion was over complicated.

IIS returning 403.1 error

Simon Baynes — Mon, 16 Feb 2009 22:33:11 GMT

Had a strange issue with IIS this afternoon at work that I thought I would share.

I am in the process of migrating an application from Apache to IIS and during the testing I discovered a strange error. When trying to return an HTML file IIS was returning a 403.1 status code and moaning about execute permissions.

I spent a while checking all manner of security permissions only to discover that it was because the file was in a folder whose name ended in '.com'. Apparently this causes issues because IIS thinks that you are trying to execute a file with a '.com' extension.

Sadly I had to recode my architecture to change the folder name as there did not seem to be a work around.

Anyway hope this helps someone someday who is scratching their head!

Using Java to Optimise Looping Over Lists

Simon Baynes — Sun, 16 Jul 2006 10:24:23 GMT

Once you have been using ColdFusion for a while you will have undoubtedly had to loop through a list. Now this is very straight forward operation and one where you rarely have to consider performance. However, if you are reading in a 10,000+ row CSV file this iterative process can be very slow. Many of you would do something like this:-

Now there is essentially nothing wrong with this code, however this processing is not optimal performance-wise. Fortunately you can access the power of the underlying Java to up the performance.

I have seen this provide enormous performance gains.

SQL Optimisation Tip

Simon Baynes — Tue, 16 May 2006 13:51:45 GMT

There is a saying 'There are many ways to skin a cat' this is rarely as true as it is with optimising SQL.

However, I have a tip for you for SQL Server. You can specify that you wish to do a 'Dirty Read'. This means that you want the select to go ahead without worrying if there are locks on the table / row because of inserts, updates, and deletes. So if a row that fits your criteria is in the process of being deleted it will still end up in your results.

So obviously this is not a solution for every query but it can really relieve some pressure if you do not mind the draw backs.

SELECT *
FROM myTable WITH (NOLOCK)

It will also work when you do joins but you must include the WITH (NOLOCK) command after every table declaration.

SELECT *
FROM myTable a WITH (NOLOCK)
INNER JOIN myOtherTable b WITH (NOLOCK)
    ON a.ID = b.ID

Promotion

Simon Baynes — Thu, 20 Apr 2006 22:40:04 GMT

I got promoted yesterday. I am now the Senior Developer at Haymarket Publishing. Well done me!

CGI Scope Fun and Games

Simon Baynes — Wed, 29 Mar 2006 15:21:44 GMT

I recently discovered that if you use to output the CGI scope then it doesn't actually display all the keys in the CGI structure. It actually displays a defined list of keys which I unfortunately only realised after about an hour trying to work out where the variables Apache was supposedly setting were.

Very annoying.

Using Array Notation in ColdFusion

Simon Baynes — Wed, 22 Mar 2006 18:53:30 GMT

Referencing dynamic variable names can be tricky, unless you are aware of a few basic ColdFusion concepts.

I see things like this:-

  
    
    #temp#

and it drives me insane. It is so unnecessary not to mention messy.

Here it is again but using array notation.


    #form[iFormKey]#

Now this is not only optimal it is also clear. If you find yourself using the evaluate() function you are either going about it in the wrong way or you are trying to cheat ColdFusion into doing something that it doesn't want to do.

Also bear in mind that barring a few exceptions all ColdFusion variables are in a struct.


    myVar1 = 1;
    myVar2 = 2;
    myVar3 = 3;
    myVar4 = 4;
    myVar5 = 5;

    for (i = 1; i LTE 5; i = i + 1) {
        // now we use the fact that by default any non-scoped variables are put in the variables scope to our advantage
        writeOutput(variables["myVar" & i] & "
");
    }

So as you can see array notation is very powerful and clean.

Default Proxy for ColdFusion

Simon Baynes — Fri, 17 Mar 2006 15:09:11 GMT

Ever had to develop an application that used cfhttp? Ever done this on a machine that is behind a proxy?

If the answer to those two questions is yes then you may have written some bung code like this:-

This is totally unnecessary as with some jvm.config arguments you can set a default proxy for your ColdFusion instance.

-DproxySet=true -Dhttp.proxyHost=255.255.255.255 -DproxyPort=80#