Admiral vs Google

It’s not hard to find stories about how Google is such a great employer, they have amazing canteens, free gyms and a generally all round “cool” attitude. I have never worked for Google but I must admit that I think it’d be pretty interesting but also pretty tiring. They do hard and complicated work there and though I’d enjoy it; I’d get burnt out pretty fast. Now we come to my current employer, Admiral Car Insurance.

Admiral has a very good policy which can best be summed up as “Happy employees produce higher quality work”. It’s easy to find a company that emsays/em it subscribes to that but this is one of the few cases where I’ve seen proof of it. I could list how the vending machines are very cheap, everybody from sales staff to managers dress casually and how the managers are really nice people but I think I can better sum it up with one line.

There is a strongMinistry of Fun/strong. That’s right, groups of people take it in turns to organise competitions, trips and general coolness. Before you worry that I’m giving away company secrets; it’s all located on their website a href=”http://www.joinadmiral.co.uk/Culture.aspx”here/a.

I must say that when I left university a year and a half ago with my lovely degree in Computer Engineering I never thought I’d be working as a salesman for a company quite as cool as this.

I have a job and will be starting it on the 9th of November. This means I need a car and on Saturday I found one (with much aid from my Dad and brother). There are pictures of the front, side, rear and other side. My ever helpful uncle Brett has suggested a slight alteration to the paint scheme, any thoughts?

I know I said I’d post up a database sync script but on the basis that this blog has low readership and that it was actually not as cool as I thought it was I changed my mind. I looked at it and realised that most of it is really simple stuff; most of which is tied into other parts of the program. Today I found that I will be starting a new job in early November and on the basis that I’ll meet people and do cool stuff each day this blog may find itself becoming more interesting and updated more frequently. Who knows, maybe I’ll be able to post stories of my attempts to woo young ladies, probably not though…

You may notice that I’ve not posted a caching article yet, that’s because I’ve simply not needed to. I have cached only a few functions and have achieved a query duplication rate of only 1.5-3 times. Sure that’s not good to have on a public web server but this is my computer and the script runs approximately once a day, it’s simply a matter of the cost-gain ratio. But fear not, I have something related to write about and that’s the underlying “problem”.

As I said, this is a 23 thousand lines of code (not counting those comments and empty lines) program, actually it’s 24.8 now because I added some features but that’s not the point. The point is that it’s not a small script that’ll do one or two simple things, it’s a pretty big application and I knew from the start it would be. I designed it knowing these things, I knew these things because I previously wrote something very similar in PHP, I knew what I wanted to avoid. But enough about how I apparently know what I’m doing, lets get to the nitty gritty.

Modularity
Rob2 is very modular. Modularity means that I reduce code duplication and that I can copy+paste+edit code from one part and use it in another. Modularity brings with it the curse that things need to slot together which means that some things will be sub-optimal. A good example is that several places in the code can run a query that goes something like this:

SELECT * FROM armies WHERE id = X

Where X is a number, part of my initial optimisation was to find these things and make them call a complete list of the table as it’s cached.

Laziness
Like any good programmer I’m lazy. Like any good programmer; this can cause problems for me. Since I use a hand-rolled Database layer I’m often selecting * from my tables and often from the entire table. Normally this is good and I then cache it so that it runs real fast. In many instances however I will call * from the entire table and not need it. Worse yet I’ll call it for the whole table when I need only 1/20_th of the table! In a situation where resources are limited this is clearly not acceptable but like I said, this is not one of those situations and the code is so much easier to follow because it all follows the same design.

It is however not all bad; there are some things that I’ve done well. Some things are common sense while others might not be (I’m not common and may or may not possess sense).

Indexes
Most of my tables have indexes. The trick is to put the index on the field that you apply a WHERE statement to, not the field that you’re trying to get at. I didn’t know this at first and thought that my queries were loads faster because I used indexes! Indexes provide a saving in speed for a cost in space, I have 58GB of space left so I probably have a couple of indexes I don’t need.

IDs
I store stuff by it’s ID; not by it’s name. Searching through an integer field is a lot faster than searching through a length 40 varchar field. Better yet, the integer has no reason to change but a user can change their username and that causes all manner of “fun” for the developer(s).

Duplicate data
I intentionally duplicated one of the fields in my database and for a good reason. If I didn’t then at several points there would have been a 3 table join and several queries that don’t use a join currently would have needed one. It’s data that won’t change and it’s an integer, thus it doesn’t come into conflict with the main reasons to ensure data normalisation is applied.

Finally…

Lack of obsession
As a developer it’s very easy to focus on one thing to the exclusion of others, premature optimisation being a pitfall I fall into if not thinking carefully. In this instance I implemented a basic cache system and left it at that until a few weeks ago. By not obsessing about the speed of my program I completed it faster and then when I did optimise it a bit, I continued to not obsess and have since added new features to it. If a script runs 10% faster but it took you 2 days to make the change then it’s probably not worth it. I say probably because as I’ve been so happy to point out, my situation is not your situation, there are cases where that 10% will be the difference between success and failure.

For my next trick…
I will be sharing the database checker that I wrote for Rob2. It runs through the data types and matches them to the table, automatically adding/removing columns and indexes.

I know I said that I’d develop on that caching system but I’ve not had a chance! I have something more interesting anyway; yes it’s hard to imagine anything more interesting than a caching system but it’s true, there are things more interesting than caching (granted not many). I had a programming-exam-interview… thingie.

The test itself was not overly complex, I had an hour to complete as much of a simple yet tricky task, a good understanding of regular expressions will mean that you can do it, you just probably won’t have time to do all of it. The test itself is not the interesting part though, screen scraping it as anybody knows, boring. The part I want to talk about is the fact that there’s a test and the way it was done.

In theory the test will do the following:

1. Prove that a candidate can actually program as listed on their CV
2. Show exactly how well a candidate can perform at a task that will be a key part of their employment at a company
3. Show how good the candidate is at working on their own

I want to explain how this is both good and bad.

Bad
Bad comes in two parts, exam and my selfishness. As with any exam, if I was having a bad day, if I’d not gotten much sleep (which incidentally I didn’t) or any number of other things then it’d have counted against me, it’d have made the test unfair and lowered my chances of progressing further. Obviously this is pot luck and nothing specific against me so by the very same logic I can have good conditions and all other candidates can be so ill they can’t even check their email.

The second part of the bad is my personal selfishness. If I am below average compared to the other candidates (and statistically there’s about a 50% chance I am) then it’ll count against me as would any test of skill. The test was pretty open, I was allowed to use any language I wanted and to output the data into any one of several common formats so that was okay. Again this can go both ways, if I’m above average then this obviously increases my chance of progressing further so is it really bad?

Good
There’s one good thing about it but it’s a big good thing and I think it outweighs the bad by a lot. Assuming all those that work there undergo a similar test it means that the people I work with will be better qualified and if I do slip up (and everybody slips up) then they’re more likely to be of help and it also means that I’m more likely to be able to learn from them and converse in the same terms.

I think it’s a great idea and will be thinking more highly of any companies that employ such a scheme.

The program is written in python, twenty three thousand lines of code (not counting empties and comments), it’s source weighs in at 3.4MB. It runs as a web app and connects to a postgreSQL database for it’s data storage, it’s got a caching system to reduce database load and all around is the best program I’ve ever written. I knew from the outset that it’d be hard to optimise as everything is very small and modular, everything calls a lot of other functions and often had to deal with a large amount of data.

This led me to three assumptions:

1. Any optimisation would produce only small gains as each script was calling many different functions, not the same one over and over again
2. 64 seconds for the script execution time was quite okay considering the size of output
3. The caching system meant I’d not be overloading the database

The first thing I did was split the script into sections and time each section. A print of the timings showed that some sections took far longer than others, this was expected as some sections had to handle a lot more. On a whim I decided to try using the cProfile library I was confronted by several pages of functions each of which was taking up less than a tenth of a second of the total time. I also found that one of the functions was sucking up I think about 50 seconds, it was the database execute function.

I hacked into the database script something to record the queries in a list and then used the set() function to find out how many unique queries there were. 40,000 queries in total, 3,000 of which were unique. It seems as if my caching system was not working, but it was, the caching system that I’d written was just fine. What I wasn’t caching were smaller queries that often went “SELECT name FROM table WHERE id = %d”.

Caching these has knocked it down to about 5000 queries in total and the total script execution time to 10. What I want to share here are two things. Firstly you need to throw out assumptions, you never know if your program is wasteful until you look at it. Secondly is the code I use to handle the cache, it’s something I whipped up in just a few minutes so I’ll post an improved version in a few days.

import collections
function_cache = collections.defaultdict(dict)
 
def my_database_function(arg1, arg2, recache=False):
	f = 'my_database_function'
	a = "%s,%s" % (arg1, arg2)
	if a not in function_cache[f]:# This is to prevent us getting key errors
		function_cache[f][a] = {}
 
	# If we have the cache and we're not being told to recache, send them the cache
	if function_cache[f][a] != {} and not recache:
		return function_cache[f][a]
 
	function_cache[f][a] = do_our_query
 
	return function_cache[f][a]

As I said, this was something I whipped up in the space of a few minutes so there’s probably a better way to do it with decorators and a dedicated class. I’ve posted it up like this because I’m happy to show that my code isn’t perfect first time around and because somebody may provide some useful input before I start on the more polished version.

No I’ve not blogged in a while , I blame it on Brett for uploading baby photos of my to facebook, he’s successfully scuppered any chance of me finding a girlfriend that uses facebook. It’s been a fun few days with the marriage of Luke and Becca Orpin. I’m not going to regale you with in depth explanations of my poor dancing skills, amazing courtship abilities (stymied of course by the aforementioned pictures) and bottomless pit of a stomach.

I stayed at Becca’s parents, it was really nice as they gave me a lift from the station both ways, there’s no way I’d have been able to attend if it was not for them. Both Luke and Becca were pretty good at handling the stress involved in setting up.

Funny and memorable things include:

• David (Luke’s dad) trying to outwit me in a verbal battle.
• Ceryn explaining why she now has a car with racing stripes.
• Becca trying (and failing) to convince me that she was not trying to set myself and Ellie up, despite making us the only two from our group on the table. Poor Ellie.
• Lorna trying to wind Greg up “I have more pictures of Tei than of Greg!”.
• Luke and Becca realising after 3 minutes through their first dance that they still had 2 minutes to go and trying to call others to start dancing.
• People from my table telling Luke and Becca that they were really glad I was at their table because I am “so funny”. I think they may have been serious!
• Me being used as a tanoi/announcement system.
• Graham’s cry of “Noooo” as Brierly tried to catch the bouquet.
• On the way home finding a Russian girl on the train that has wanted to go to Cardiff for years, I wasn’t aware it had such a reputation.

I’ve missed out some things as they were a “you had to be there” sort of thing but ultimately I had good fun. Note the complete absence of pegs, cookies and potatoes.

So there I was sitting at my computer browsing reddit when I found a personal ad in graph form. It occurred to me that maybe the cookie theory isn’t the best idea, for a start I rarely venture out of the house. I should forget trying to be “normal” (which is another word for boring) and start approaching the problem with a bigger dose of logic than I had previously.

So I made a graph. I have however not simply copied the idea that inspired this post, I have made two big changes. Firstly the stuff on the left is all stuff that reflects me in a good light, the stuff on the right is not so good, things that maybe you might hesitate to share about yourself. The second change is that I’ve left the right hand side of “her” side blank.

As you’ve probably guessed this is more an exercise in logic and reasoning than an actual attempt to woo the woman of my dreams. It does I hope raise an interesting question, we can pick our own achievements and to a certain extent limit our flaws, but we can never really get rid of our flaws or we stop being who we are. Thus when looking for someone, should we focus on what they chose to be or what they are but wish they were not? Should we focus on what we like or what we don’t?

On Saturday it was Martin’s birthday. That’s great and all but the party was on Monday. We first played Rock Band and we each had a go on each instrument. I was so bad at the singing that everybody said not to sing again, they tried to be polite about it (except for Sam of course but he’s just rude) but I’m well aware that I cannot sing in tune, tempo, key or rhythm. I was unable to play the drums properly as I have a cast on but I wasn’t too bad at the guitars!

Then we played rabid rabbits on the wii. This was hilarious (mostly because the other 6 people were all couples and had to share, I shared only with myself), the game consists of many small minigames. The best of which was a toboggan thing, you had to sit on a wii fit board and lean back to go faster. Gwen and Ceri seemed to forget you could go faster while Charlotte leant back a little bit. If you lean back too much a big warning sign comes up to remind you that you could hit your head on the floor. It was a slalom course and not only did I miss 90% of the gates but I managed to have the warning on the screen for almost the entire time, I also won that round.

In the evening we went to Nandos, it’s a chicken restaurant and really tasty, I recommend it.

This is intended to be a status update on Rob 2 for those that are eagerly awaiting news of when it’ll be complete.

GUI
The first function of Rob is as a database and to be a way for me to interact with that database. Thus there is a GUI (I’ll make a screencast of it one day) and that part is currently 80% complete. I still need to add in the Wars GUI which has become a lot more complex since the new army system was thought of.

Orders
The second function of Rob is to download then execute orders. This is still very much not complete. It’s much harder this time around because I’m abstracting database from rules (so I can run a game other than WoA using Rob 2). So far Construction and Research are complete and the others will soon follow as I get more used to the new way of doing things and can copy+paste more code each time.

Processes
This covers things such as making the Map and TO. Rob can make the Map but not the TO. He can make most of a TI but not all of it.

I still have no solid ETA for Rob 2 but I suspect it’s a month or two away from completion (all depends on how my days go). If anybody has any specific requests for me to include something in the screencast then leave them in the comments.