Your 2 year old knows how to grab your smart phone out of your pocket and figure out how to play a video. As technology becomes more intuitive to use and more integrated with everything we use on a daily basis, designing a user-friendly experience becomes easier to do. All this exposure that we have to technology teaches us to expect objects we interact with to act in a certain way and when we like how it works, all we have to do is copy it to create the next new product. But what about when we don’t like how it works? This is when you can learn from your grandfather that can’t figure out how to place a phone call on the same smart phone that your child is using to ask Siri the answers to his homework. Sometimes it is great to jump on board with the newest trend, but sometimes it’s beneficial to think about why a trend has become popular. This gives you a chance to take a step back from technology and consider how it fits in to the world around you, and is something that life experience will compel you to do more often.
I learn from my experiences as a mom and sometimes apply the same techniques when brainstorming for a new product: How can I simplify my instructions enough that a child would understand it? How can I design something that will be used in the way I intend? Will the outcome be what I expect? (If I pack a lunchbox with a whole wheat sandwich, a vegetable and cookies, can I really expect that it will be returned to me with any leftover cookies?) How can I clean up this mess in a way that it won’t happen again (ok, that one I’m still working on…)? We pay close attention to the feedback we get from the people who use our products. We use this combination of life experience, research, data and exploration to constantly make them better with each iteration.

Each day is a chance for something new as a developer and most days that is the case. Feedback drives you to improve and create things that give better value than it’s previous version. This pushes you into new methods and techniques for accomplishing these goals. I may know what I want to do, but I also need to know the best way to do it. This could range from displaying information in a pleasant manner or it could be making a task more user friendly. The end result will have all the benefits of the original with that extra dash of speed, scalability, or delightful appearance. Once you’ve accomplished this, it is time to do it again. Except this time you have one more angle to view your problem from. Now it’s tomorrow and you’re back in the same position as before. “Show me more!” or “I would also like it to do this” are exciting to hear, because that is what was said a week ago and you accomplished it.

New things will only ever be new once! Obvious I know, but it is easy to forget that when your staring down a path where the end isn’t quite clear. Remember that the unknown from last week is that shiny feature that everyone is using and that any task that comes your way will add value to you and those that depend on you.

As a developer, it’s very easy to get caught up in the minute details. Many issues take lots of time to delve into very specific, intricate pieces of code. Although this is the case most of the time, some things only require the simplest, most obvious of fixes. Well, these quick fixes are almost always going to be unseen when we have our blinders on for complex causes.

Something I, and probably many other developers forget often, is to step back and look at the bigger picture. I had a case recently to figure out why some lines didn’t have entries in a certain table. I checked the logs for the lines, where they were changed, when they were changed, what configurations were edited, etc. Had I stepped back and looked at the table that held the most simplest of line configurations, I would’ve seen that they were test lines and didn’t even particularly matter.

In this instance, looking at the bigger picture would’ve saved valuable development time. So how do we know when to look bigger? Well, I think when it comes to development that it never hurts to take a few minutes and check the basics. Instead of diving straight into checking the database and page setup for why a variable isn’t defined, maybe just check the syntax where it’s set. Countless times i’ve come to find that there’s just an extra space or special character getting in the way.

There was a good example of this scenario recently. Given this issue, what would be your steps to troubleshoot?

Google.com is not rotating

1. Check the website (no rotation)
2. Check the WST pool configuration (all good)
3. Check the website for our script (placed)
4. Check for console errors/correct placement (all good)

What, then, could be the problem? Well we all just assume that the pool is set up under Google.com. If you put that in a string it would look like this: ‘Google.com’. The overarching problem in this case was the string: ‘ Google.com’. One space before the URL broke everything. This is a quick, basic fix.

So whenever you have a problem to look at,  try to keep in mind the simple things. Before getting too immersed in things, just look at the big picture and you might just find a fix there.

So you’ve got this really cool website. It’s an analytics platform with a bunch of snazzy charts and graphs, elements that fade in and out, and animated icons that really make your site pop. You also have a huge database full of rich data that enables you to produce the most detailed reports in the industry. Because of all this, you’ve been gaining popularity among users around the world and have been getting some attention from some pretty big players. What you may not know, is that you have also been getting attention from another crowd: the malicious user.

Who is this malicious user? It could be anyone. It could be someone looking to steal sensitive information, or it could be someone just looking to mess something up and ruin your day. You won’t know who they are, or where they’re coming from, so your website needs to be prepared to handle anything that can be thrown at it.

I will outline a few things you can do to ensure that your website is secure against outside attack. This is by no means an exhaustive list, and I will not go into painstaking details about any of these methods. This will just be a primer to get you thinking about security and possible methods of attack.

Input validation: Don’t trust user input. Period. Validate any input you receive from users. Whether that’s a form fill, a POST request, or any other information provided by an outside source. Make sure what you are receiving is what you are expecting. For the best user experience, you would validate within the browser, most likely through javascript. If you do this validation, make sure you do not rely solely on it. Do server-side validation also. Many server-side languages have built-in validation methods that make this fairly simple. If you come across any input that isn’t expected, stop processing that certain request and direct the user to a safe place.

Cross Site Scripting (XSS) Attacks: This goes hand-in-hand with input validation. Cross Site Scripting (or XSS for short) is a when a user injects a rouge piece of code (usually javascript) onto your website and the script executes without your knowledge. There are three types of XSS attacks: Stored, Reflected, and DOM-based. With all three types, the best way to combat these attacks is to escape any user-provided input before the input is loaded into a page. What this does, is turn any special characters into either HTML entities or their unicode equivalents. This prevents the browser from interpreting the special characters as their special meaning and instead displaying them to the page as string literals. Many server-side languages have built-in methods for escaping input into either HTML entities or unicode characters.

Database Injection Attacks: Also going along with input validation is preventing Database injection attacks. You should especially think about this if you are using SQL or any other database language that allows multiple commands to be run within the same batch. If you do not validate and sanitize the input provided by a user, then you could be open to unexpected updates, dropped tables, rouge inserts, or any other kind of behavior you are not expecting and do not want. This could in turn crash your website, or destroy the integrity of your data. A user could place the following code into the Name field of a form: ‘Bob; Drop Table users;’ If you just take this form data and plug it into an Insert statement on your SQL database, you just dropped your entire user table (if you named that particular table ‘users’) and now no one would be able to access your website with their credentials. This leads immediately into another security concern: don’t share or display your database structure to anyone that doesn’t need to know those things. Once someone has access to your table names and fields within those tables, they could wreck your entire database if they have access to an unsecured form.

I only scratched the surface of how to make a website secure, but the information here would be a good place to start on the journey to making your site impenetrable.

So this is going to start by comparing systems to a growing child’s shoes. The first thing you search for when shoe shopping is something that you like/suits your needs. For example, you would not buy flip flops for hiking. You also should not buy $100 shoes when you only had $50 to spend on them. The same things apply to the servers and hardware needed for the infrastructure of the systems.  You have to work within a budget when buying servers, computers, network hardware. In most cases most bang for the buck is sought after. Some brands that may not be liked of viewed as high quality and are avoided in the selection process. There are some nice to haves like binky shoes or quad Ethernet ports that probably are not needed.

The next comparison to be made is when you buy a new pair of shoes for a child you need to allow room for growth. When buying network hardware, it is good practice to try to enough space to grow. Ideally the hardware would before be replaced before it can no longer keep up with the load that is pushed on the system. Then again sometimes the systems grows much faster than expected and this is both a good and a bad thing. The bad part being that the system fails to keep up with the need and the good part being that you are doing enough business to need to upgrade. There is usually an expected end of life for hardware.

Going back to the right shoe for the occasion, you need to have the right hardware for the job. Certain types of servers are need for web traffic and certain server for databases and firewalls are different as well. A database server should have a good processor set up, a lot of storage with variation in the speeds of the drives.  An SSD is good for the C drive but not necessarily needed for the storage as SSD’s rise exponentially in price as storage goes up.

The last comparison to be made is recognizing with the kids foot herts because their shoe is too small for their foot. In the systems world, there are issues that are glaring and some that are not. Some of the glaring issues are the database getting locked up on queries being ran. A few of the things that could cause this are too many requests on certain tables, databases or even on the server itself. Sometimes, a database gets used so much, that it needs its own server. In the instance of a webserver, if the database server has been ruled out as the blocker, sometimes the load needs to be split between multiple web servers or there is an underlying issue or issues with the webserver. In the end, there should always be an expectation that hardware will need to be replaced because of upgrade, hardware failure or plain expansion.

What is clean coding?

First and foremost, what even is clean code? At its core, clean coding is writing code that is meant to be read by people. This is an important idea. If you take only one thing away from this post, it should be that when you write code your audience are the people who will read it in the future.

Why code cleanly?

The computer is much better at reading code than people are. As long as the code is syntactically correct and does what it’s meant to, there won’t be any problems on the computer’s end, but people have to read that code. If the code is essential to your business, it’ll be read repeatedly as your business adapts and evolves. If the code was haphazardly thrown together in a hurry, every person who goes to read the code is going to have to slowly muddle their way through it, and they may not even understand it when they go to change it! This wastes countless people’s time and introduces bugs. Code like that is high in technical debt. From wikipedia:

 
In short, technical debt is stuff that you should’ve done, but you either forgot to do or decided to do later*. Technical debt can and has killed companies. Clean coding is your front-line defense against technical debt. It won’t eliminate it, but it’ll help greatly.

How do I code cleanly?

There are many suggestions elaborated thoroughly and with plenty of examples in Robert C. Martin’s book Clean Code, so I’ll just give the abridged version.

Choose names carefully

Names should be descriptive. Variable names probably should be nouns. Function names should start with an action verb and be “unsurprising”. That is: if you see the function name “get_all_users”, when you read its implementation you won’t see anything that makes you double-take.

Obey the Single Responsibility Principle (SRP)

If you don’t know what that is (shame on you**), here’s wikipedia’s definition:

 
This extends to functions as well. They should be small and do one “thing”. Complex behavior should be composed out of simple parts.

Write clear, informative comments

Comments should be rare. Your code should speak for itself, except for when it can’t. That’s where comments come in. Your comments should clarify complex code and warn of dangers.

Format your code consistently

It doesn’t matter whether you prefer snake case to camel case or you like your opening braces on the same line as the function or on a new line. Be consistent. Preferably, your entire codebase should be governed by the same style guide.

Wrap-Up

Those are the most important suggestions. The only one that doesn’t directly relate to making the code easier to read for humans is obeying the single responsibility principle, and a major effect of the SRP is shorter functions and classes. Remember: your audience when coding is the people who will read the code in the future!

* Later almost always means never.

** More on the amazing value of the SRP can be found in the article “KISSing SOLID Goodbye” in Overload 122.

It’s a well circulated science “fact” that every face you see in your dreams is one you’ve seen before, some time in your life. It could be your best friend, your greatest enemy, or a total stranger you only saw once in a crowd at the mall, but it’s still a “familiar” face. Never mind the fact that this claim is essentially impossible to test; the idea of it is powerful. On a fundamental level, it taps into a basic human understanding that faces are important.

Tech companies know this. That’s why Google tried (and failed) to clean up YouTube comments by forcing people to comment using their real names and faces from Google+. It’s why Facebook works as a business: seeing your friends’ faces makes it easier for you to offer up all kinds of information about yourself that you’d never give to a giant company otherwise (after all, Facebook isn’t asking for it, your Aunt Rhonda in Minneapolis is). At Century Interactive, we’re no different. Faces are important, both in our internal and external products.

Internally, we just rolled out a new tool we’ve dubbed LUNSH. Not only is it amusing to attempt to pronounce properly, it’s a fun and interesting way for CI employees to get out and see each other’s faces and learn about their coworkers.

No project is complete without a bizarre acronym.

Six days of the week, the tool shows the Opt-In Screen. Here, our employees can say “Yes, I want to sign up to LUNSH!” and can do it with a single click. They’ll get to see who else has opted in and a few fun stats (because when you deal with as much data as we do at CI, you enjoy finding new ways to use and display it) and that’s it for those 6 days. Day 7 is completely different: Day 7 is LUNSH Day. On Day 7, our employees see a new screen, populated with LUNSH Groups. See, on LUNSH Day, everyone who opted in gets sorted into a random group with up to five other LUNSHers. All of our data is stored in a SQL database, which unfortunately doesn’t include some mystical built in function for randomly sorting a bunch of data. To get these random groups made, each opt-in record is assigned a random number (which IS a built in function). On LUNSH Day, we sort the table using this random number, giving us our random sort! Then we just cycle through this week’s records and pop a record off the top of a list, throw that person in a group and repeat until we’re out of records! Each group then gets together, picks a restaurant, and enjoys lunch (er, LUNSH) with their coworkers. It’s a fun and easy way to meet up with employees in different departments that you may never see in an average work day and connect with the people you share an office with 5 days out of the week. If you’re lucky, the CEO may wind up in your LUNSH group! (Dress nice on LUNSH Day, just in case).

Before long, you’ll be seeing LUNSH high scores on resumes worldwide.

We have the same philosophy when building our external products. Car Wars, Service Setter, and Patient Pursuit users have probably noticed the images in the staff pages. It would have been easy (certainly easier to code!) to just list names and call it a job well done, but it was a deliberate decision to include staff photos in throughout our products.

Look at all those smiling, placeholder faces!

Like Google’s attempt to make YouTube commenters more accountable, adding faces into our products helps maintain accountability and make what would otherwise be a dull stats table into a more exciting page. With staff photos, it’s easier to attribute the statistics you see (whether it’s an individual call or a CRISP score) to a person or a coworker or a friend. It makes it easier to recognize good calls and praise a job well done (instead of “Wow, we had a good call” you think “Wow, Stephanie had a good call!”) or recognize a difficult call and work towards improvement. When your staff uses one of these products and sees their calls and metrics attributed not just to their name, but to their face as well, it’s easier to take ownership of that call and recognize it as a personal accomplishment or area for growth. Even if your staff list still displays our default photos, it’s easier to think of a person behind all that data when you see those faces; people trying to provide a great experience to the caller and striving to be as professional and helpful as possible. This effect is even more powerful if you have already uploaded staff photos, but even those CI placeholders (we don’t blame you for keeping them, they’re some good lookin’ CI team members!) can leverage the power of a friendly face.

You may think CI is all about the calls and data, and we are. There’s a lot of power in data and the connections you can make with it. However, we like to think there’s something we work with that even more important and integral to our business and helping grow yours: faces and the people behind them.

This is the second post in a series of posts on Network Programming.

For the first part of the series click here.

In the first post we introduced UDP and TCP sockets, showed how to create server sockets, bind them to an address, and showed what happens when a message is too long to be read in a single call to socket.recv.

Recall that the first time we had our client talk to our server we had it send “the rain in spain stays mainly in the plain” which is only 43 characters long, but the server was listening for 100 characters.   This is because (per the documentation) the 100 we are passing into the recv method is the maximum number of bytes to read.

So how do we receive a message that is more than 100 bytes?   It’s easy we make additional calls to socket.recv.

#!/usr/bin/env python

from __future__ import print_function
import socket


def main(*args, **kwargs):
    print('creating server socket')
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    print('done. now binding to address')
    server_socket.bind(('0.0.0.0', 54321))
    print('done. now listening for connections')
    server_socket.listen(20)
    try:
        while True:
            client_socket, client_address = server_socket.accept()
            print('new client connected from: ', client_address)
            needs_to_keep_reading = True
            while needs_to_keep_reading:
                content = client_socket.recv(100)
                print('#### new data received ####')
                print(content)
                needs_to_keep_reading = len(content) == 100
            print('#### finished receiving data ####')
            client_socket.shutdown(socket.SHUT_RDWR)
            client_socket.close()
    finally:
        server_socket.shutdown(socket.SHUT_RDWR)
        server_socket.close()


if __name__ == '__main__':
    main()

So here we keep reading while we receive exactly 100 bytes.

Great!!!

Unfortunately, this won’t work in the real world. What we didn’t mention earlier was that we can’t control how many bytes are recieved. Even if the client on the far side does indeed send exactly 150 bytes in a single socket.send command, that doesn’t mean that two calls to socket.recv(100) will return the first 100 bytes followed by the second 50 bytes. This is because those bytes may be split up along the way as they travel throughout the internet.

This sample client program is contrived, but shows that you can’t count on always reading the maximum 100 bytes, even if more than 100 bytes are headed your way.

#!/usr/bin/env python

from __future__ import print_function
import socket


def main(*args, **kwargs):
    client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    client_socket.connect(('0.0.0.0', 54321))
    total_stuff = 'banana'*100
    first_send = total_stuff[:20]
    second_send = total_stuff[20:]
    client_socket.send(first_send)
    client_socket.send(second_send)
    client_socket.shutdown(socket.SHUT_RDWR)
    client_socket.close()


if __name__ == '__main__':
    main()

So crud. How do we know when “the message” has ended.

This is what network protocols are for. Before we talk about what a network protocol is, we should dig in to understand what a protocol is. I think the simplest example of a protocol is Morse Code. Morse Code was a simple agreement between all parties using Morse Code on how to represent letters and numbers. It also said how to separate words (seven units of silence indicates the separation of a word). However, because it had no way of representing a period or other punctuation, there was no way of indicating the end of a sentence. That’s why, when ever you see people doing Morse code in historical TV shows they are always saying STOP, blah blah blah STOP.

The word STOP was used informally as a means of indicating the end of a sentence.

Network protocols behave the same way, in HTTP a “message” is either an HTTP request or an HTTP response but both end with two carriage return line feeds CF LF CR LF. So and HTTP server must continue to read on the socket until it comes across two carriage return line feeds. But what happens if that sequence of characters never occurs? Well, that probably is an indication of either a buggy or malicious http client (or server) and typically server implementations might limit how large a message can be (if the protocol standard doesn’t explicitly state how to handle this type of situation).

There are two main ways of indicating the end of a message. One would be to state that all messages are of a certain length. The other would be to implement some sort of delimited indication that the message has ended. Using STOP with morse code and two carriage return line feeds imply that these protocols are both delimited.

Naturally, a delimited protocol is more flexible than a fixed size one, but there is no doubt that a fixed length protocol is easier to implement (no need to scan the content of the message to seek for delimiters).

So we’ll finish off this post by creating a simple echo server and echo client. The echo server will accept client sockets and it will read in as much data from the client as possible until it gets to a colon character (:). Once it sees the colon it will send back whatever the client sent (including the colon) back to the client. Once the data has been sent it will start over, reading more data from the client until it sees another colon and sending that back to the client. The server will continue in this fashion until the client closes the connection.

In our client we will allow the user to enter as much data as the user wants then send the data to the server. After it sends the data to the server it will read everything back from the server and then ask the client for more info.

Now before we get coding on this we have to address the scenario on the server when the client disconnects. How do we know when the client disconnects? From this article in the python documentation, socket.recv returns zero bytes it means that the other end has closed the connection.

So here’s our server program now:

#!/usr/bin/env python

from __future__ import print_function
import socket


def main(*args, **kwargs):
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.bind(('0.0.0.0', 54321))
    server_socket.listen(20)
    try:
        while True:
            client_socket, client_address = server_socket.accept()
            handle_client(client_socket)
    finally:
        server_socket.shutdown(socket.SHUT_RDWR)
        server_socket.close()

def handle_client(client_socket):
    socket_is_alive = True
    while socket_is_alive:
        complete_content = ''
        needs_to_keep_reading = True
        while ":" not in complete_content:
            current_chunk = client_socket.recv(100)
            complete_content = complete_content + current_chunk
            if len(current_chunk) == 0:
                print('---the socket died---')
                socket_is_alive = False
                break
        print('---got full message---')
        print(complete_content)
        print('---end of message---')
        if socket_is_alive:
            print('---sending the echo---')
            client_socket.send(complete_content)
            print('---done echoing---')


if __name__ == '__main__':
    main()

and here’s our client program.

#!/usr/bin/env python

from __future__ import print_function
import socket


def main(*args, **kwargs):
    client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    client_socket.connect(('0.0.0.0', 54321))
    data_to_send = ''
    try:
        while True:
            data_to_send = data_to_send + raw_input('enter more data:') 
            if ':' in data_to_send:
                client_socket.send(data_to_send)
                data_to_send = ''
                print_received_data_from(client_socket)
    except KeyboardInterrupt:
        client_socket.shutdown(socket.SHUT_RDWR)
        client_socket.close()

def print_received_data_from(client_socket):
    complete_data = ''
    while ':' not in complete_data:
        complete_data = complete_data + client_socket.recv(100)
    print('got data back')
    print(complete_data)
    print('done getting data')

if __name__ == '__main__':
    main()

Go ahead and run them both and notice how the server can detect when you close the client program by pressing control-c.

But try running two clients at the same time. You should notice that the second client won’t receive any data from the server until the first client stops. This makes sense because socket.accept on line 13 of the server doesn’t get called again until the last client has been handled.

So how can we support multiple clients at the same time? Well there are lots of ways to do that, but we’ll leave that for part 3 of the series.

There’s a famous quote by Abraham Lincoln that reads, “Give me six hours to chop down a tree and I will spend the first four sharpening the ax.”  What good ol’ Abe was trying to communicate with this saying is that proper preparation is key to successfully completing a task. Not only will sharpening the blade require you to assemble the necessary tools, it also saves you work when it comes time to chop down the tree.

I often think of this old adage when I am given a new report to build or feature to develop.  Instead of an axe, the tool I have to sharpen is my brain and instead of chopping down a tree, I am building software.  I am often tempted to start writing queries and assembling data before I even have a firm grip on the core of the problem.  I remember when I first started writing software, I would start furiously typing out queries and tests only to realize I was writing code that didn’t directly fix the initial problem.

Thankfully, there have been others who suffered from this lack of preparation who studied development cycles and established best practices for proper software development.  Some of the leading software development practices today are Agile, Waterfall and TDD (test driven development).

Most of the development cycles take root in waterfall which illustrated a cascading model of processes, in which progress is achieved through flowing downwards through some core main stages: requirements, specifications, design, implementation, verification, maintenance.

The waterfall methodology is appealing because specifications and designs can be created before implementation begins.  Waterfall is great concept as long as both the problem and solution are well known.  With this flow, the waterfall almost becomes an assembly line and code can be written easily.   If Abe were following the Waterfall methodology, he would have spent the first 3 hours buying the best ax, sharpening it and looking up effective youtube tutorials on how to swing an ax.  The next 3 hours would have been spent chopping down the tree.

Although waterfall provides a very clear road map to a successful launch, what happens when the solution isn’t 100% understood before development beings?  What happens when the designs and specifications made in steps two and three end up being poor choices?   For our analogy, what if Abe sharpened his ax incorrectly or discovered the YouTube video showing best chopping techniques was horribly inaccurate?

Enter the Agile Product Development Cycle.  The Agile Development Cycle starts with user stories as the driving force behind production.  Filling in the phrase, “As a ____ user, I want to ______” gives developers a clear understanding of the end goal.  In addition to starting with a customer first mindset, agile incorporates feedback loops that include testing, bug fixes and release planning.  The agile methodology takes the best part of waterfall development but starts with a solution-first approach that challenges developers to sharpen before building, test before striking and the freedom to readjust after every swing.