Well, it happened. When applying for a job, I got sent to a Codility test. And even though I guess I already had some good practice with them, I managed to do badly in stupid ways – ways that 2 years ago I had already thought about and even taken notes on. Just, in the heat of the moment I forgot about those rules of thumb.
And in fact I think these hints should be given by Codility themselves – because if not you are practically ambushed, even more so if you didn't take the time to thoroughly explore how they work. So here are my hints-to-self.
The summary is: don't think of this as a coding interview; this is rather about getting something working, ASAP.
My day-to-day programming language for the last few years has been C (more concretely, C99 in embedded environments). And yet, I consistently get better scores in Codility's tests when using Python, in which I would say I am just a beginner. For example, consider the quantity of mental cycles that in C go into (test-wise) useless things, like all the type system and its implications: use an int or an unsigned int here? An int or an int32_t? (am I being portable, or controlling memory usage / alignment?) What happens when I multiply this unsigned integer and this double? Will doubles be too slow, will floats be too small, should I better use fixed point, is this a sequence point, etc, etc).
Think about it: if you implement a shitty naïve solution, it will fail probably performance-wise when confronted to "extreme" inputs: say, big or pathological vectors. But that still means maybe about 50% in correctness tests, and will probably be better than implementing a better, more ambitious solution – and failing to get everything right in the allotted time. Extreme case: if when the timer is over your solution isn't compiling, you'll get a perfect 0%. But if it compiles even if it only solves the most basic cases, you might get a 10% or more.
Codility allows to use printf to print debug info on your program. But if you leave that in, it will kill the performance, as they themselves warn you (I don't understand why they don't fix that, either by injecting a library at runtime to nullify the printf, or by simply adding an empty #define printf). But OK, if they don't fix it, we can. So (if I stayed in C) I would add a simple
In a "real" interview I would take the opportunity to explain how that #define works, and why for example it's better than a simple "empty" define. For that you need some experience; shows that you knows where are the pitfalls, and opportunities, of the language – which in a language as small but (absurdly) complicated as C should be an advantage.
And that's it. To me it's kind of a disturbing picture. The premise I liked about Codility was that the testing would let companies get good programmers, and programmers would get an indication about how good (or not) they are and which companies do care about it; but because of how the tests work in practice, it rather starts to sound like you have to train Codility-style to even take a look at the company / be checked by the company. To apply for an embedded-C job I have to train Python and psychic myself to ignore lots of details that make me a good C programmer - or lose to someone who did.
And in fact I think these hints should be given by Codility themselves – because if not you are practically ambushed, even more so if you didn't take the time to thoroughly explore how they work. So here are my hints-to-self.
The summary is: don't think of this as a coding interview; this is rather about getting something working, ASAP.
1. If at all possible, use a scripting language.
My day-to-day programming language for the last few years has been C (more concretely, C99 in embedded environments). And yet, I consistently get better scores in Codility's tests when using Python, in which I would say I am just a beginner. For example, consider the quantity of mental cycles that in C go into (test-wise) useless things, like all the type system and its implications: use an int or an unsigned int here? An int or an int32_t? (am I being portable, or controlling memory usage / alignment?) What happens when I multiply this unsigned integer and this double? Will doubles be too slow, will floats be too small, should I better use fixed point, is this a sequence point, etc, etc).
Meanwhile, all those mental cycles are still free to use in Python.
Another example: while starting to work on a test, I though that I would need linked lists, the plan being to move elements from one list to another list. Having already decided to use Python, I got scared; how to implement a linked list there? I had never done anything with "pointers" in Python. Should I go back to C and implement a couple of helper functions? But that would be a waste of time, even if easy; what if anything happened and I needed more helper functions, and then had to debug anything? No, that would be too much of a diversion! Maybe Python would already have something equivalent? …
… one googling minute later, I was already using a vanilla Python list (d'oh! I already said I'm a beginner, didn't I?). No need to roll my linked list in a hurry, maybe debug it, all for just a minor detail of the whole test. Again, it's easy, yes, but that is not really part of the test, so why waste the time and energy? (and IIRC, after all in the final solution I didn't use the list!)
Another example: while starting to work on a test, I though that I would need linked lists, the plan being to move elements from one list to another list. Having already decided to use Python, I got scared; how to implement a linked list there? I had never done anything with "pointers" in Python. Should I go back to C and implement a couple of helper functions? But that would be a waste of time, even if easy; what if anything happened and I needed more helper functions, and then had to debug anything? No, that would be too much of a diversion! Maybe Python would already have something equivalent? …
… one googling minute later, I was already using a vanilla Python list (d'oh! I already said I'm a beginner, didn't I?). No need to roll my linked list in a hurry, maybe debug it, all for just a minor detail of the whole test. Again, it's easy, yes, but that is not really part of the test, so why waste the time and energy? (and IIRC, after all in the final solution I didn't use the list!)
Of course this also means that Codility's test is (heavily?) biased against programmers who pay attention to lower-level details. Which in an embedded environment, should be everyone. (It's kind of a bitter irony that the story of why Codility was founded includes how shitty programmers should not be doing embedded programming in things like medical devices. Now they are implicitly penalising the careful ones!)
Alternatively, if not using a scripting language to free you from that kind of burden, just remember that the finer details of your language and context might be counterproductive here; the goal is ONLY having an answer for the algorithm at hand. So, following the types example, use any big type everywhere and do not think any more about it. After all, they will want a given O( ) storage performance, and as we know a constant multiplier caused by the type sizes won't matter. Let me repeat: they don't care, because…
2. The important thing is having a result – even, almost any result.
Think about it: if you implement a shitty naïve solution, it will fail probably performance-wise when confronted to "extreme" inputs: say, big or pathological vectors. But that still means maybe about 50% in correctness tests, and will probably be better than implementing a better, more ambitious solution – and failing to get everything right in the allotted time. Extreme case: if when the timer is over your solution isn't compiling, you'll get a perfect 0%. But if it compiles even if it only solves the most basic cases, you might get a 10% or more.
What will the recruiters do when they get your scores is anyone's guess; they could check the editing history of the test and see what you did, or they could just dismiss you without even realizing – it they have N testers they probably will only pay attention to the highest scores. (Here one could argue that they have a point to do so; and that anyway a company that doesn't stop to look at the details might not be an interesting company to work for…)
So what I would do is: if possible, implement a naïve, shitty solution FAST, maybe even typing a couple of lines in a comment mentioning the naïveness and the plan for improvement. Then, comment out the shitty solution, and implement the better solution – in such a way that if the last minute arrives and you aren't finished, you can un-comment the bad solution and at least have a non-zero score to pass the first recruiting filters.
(This is positively sad, because I have seen a couple of exercises in Codility which were kinda frightening at the beginning but turned out to unfold into beautiful solutions. And when you realize the good solution after the naïve one you only want to boast. But that's a risk. And remember that probably you have another task(s) waiting; the boasting on that task might leave you without time for the rest. Go for the low-hanging fruit! And maybe you'll be able to show off to someone more receptive in a later step. For now, you are only talking to a compiler, which won't get impressed no matter what you do.)
3. It pays off to have a debugging infrastructure with a kill-switch.
Codility allows to use printf to print debug info on your program. But if you leave that in, it will kill the performance, as they themselves warn you (I don't understand why they don't fix that, either by injecting a library at runtime to nullify the printf, or by simply adding an empty #define printf). But OK, if they don't fix it, we can. So (if I stayed in C) I would add a simple
#define printf(…) do {} while (0)… which can be just commented out when you do want the working printf. That way, in the last 30 seconds of the test, you can just make sure that the #define is active and so no printf will destroy your hard, febrile work.
4. Your real-world expertise (probably) doesn't matter, only the algorithm.
In a "real" interview I would take the opportunity to explain how that #define works, and why for example it's better than a simple "empty" define. For that you need some experience; shows that you knows where are the pitfalls, and opportunities, of the language – which in a language as small but (absurdly) complicated as C should be an advantage.
But in a Codility test this is just some trick that will help you stay afloat, instead of something you can demonstrate or even show off. It's a baseline. And hey, I can agree with that – a good C programmer SHOULD know that. But again, why is the baseline so high in C and so low in Python?
Another type-related example: in real life, performance can of course suffer a lot depending on how sensitive is the platform (hardware, OS, libraries) to the type sizes; I already wrote a blog entry some years ago about how the memory access stride in AVRs can badly change your timings. But again, for Codility just get a big type and forget about it. (And hope that their O( ) detection and time limits when evaluating your code are good enough…)
If I write a loop in a program and see that there are special cases (at the beginning or at the end of the loop, for example) I have a tendency to try to massage the loop so the special cases can be folded-in in a natural way. That also comes with the language: sometimes you can use a while(){} vs a do{}while(), for example, and choosing one or the other makes things smoother; but sometimes you are supposed to use a seemingly awkward but idiomatic construction like Python's while True: … break.
But even these choices can be used to demonstrate that you know the law of the land, the idiomatic expressions in the language and when to choose them; shows that you are following the Principle of Least Astonishment, that you write code with readability in mind – since any programmer would be able to understand some unholy goto contraption in C, or a C-style for loop in Python, but on revision, finding non-idioms might make you waste precious attention on clerical details: "why is this done like that, am I missing something?". And over and above language specifics, it can even show (off) that you pay attention to more theoretical things like having a clear loop invariant. Etc, etc, ad nauseam.
But for Codility, it probably is better to forget all of that: if you already wrote it and it's working but it's ugly, do not clean it, just move on. Make it work, no matter how Fugly it is. Add a comment maybe, for the case where the recruiter might look into the source. Or for the case when you end up having time to spare and can come back at the end of the test!
5. And yet don't get too algorithmical-idealistic.
If I write a loop in a program and see that there are special cases (at the beginning or at the end of the loop, for example) I have a tendency to try to massage the loop so the special cases can be folded-in in a natural way. That also comes with the language: sometimes you can use a while(){} vs a do{}while(), for example, and choosing one or the other makes things smoother; but sometimes you are supposed to use a seemingly awkward but idiomatic construction like Python's while True: … break.
But even these choices can be used to demonstrate that you know the law of the land, the idiomatic expressions in the language and when to choose them; shows that you are following the Principle of Least Astonishment, that you write code with readability in mind – since any programmer would be able to understand some unholy goto contraption in C, or a C-style for loop in Python, but on revision, finding non-idioms might make you waste precious attention on clerical details: "why is this done like that, am I missing something?". And over and above language specifics, it can even show (off) that you pay attention to more theoretical things like having a clear loop invariant. Etc, etc, ad nauseam.
But for Codility, it probably is better to forget all of that: if you already wrote it and it's working but it's ugly, do not clean it, just move on. Make it work, no matter how Fugly it is. Add a comment maybe, for the case where the recruiter might look into the source. Or for the case when you end up having time to spare and can come back at the end of the test!
6. Consider what the recruiter gets about you.
The recruiter gets a report in which the very first thing are summaries, in traffic-light colors and with big numbers, with the general and partial scores. So: getting any naïve result will at least avoid zeros here.
And in the probable case that the recruiter is actually someone non-technical from HR, then… do you think they will get past that? I doubt it. After all, they have outsourced the technical first step.
And in the probable case that the recruiter is actually someone non-technical from HR, then… do you think they will get past that? I doubt it. After all, they have outsourced the technical first step.
Anyway, for the case of a technical recruiter, it might happen that s/he will pay attention to the actual code if you didn't get a perfect score. So you should know that after the summary, s/he gets the history of your edits in the text boxes. So, even if you use another IDE or environment to make your programs, better copy+paste your code to Codility's window from time to time, so there is a timeline and a vision of how the code has evolved. However, as already said, I would leave the naïve version clearly visible, even if commented out. Maybe as a second function, if the first one was finished correctly.
(UPDATE: my recruiter was actually technical enough to say "the solution to the first problem was very elegant", and yet he didn't even look at the second solution because it didn't compile; he didn't care that it was almost finished, just that it didn't compile. So, once more: better get a bad but "working" solution than a good but unfinished one. Because "Worse is Better", right? ... how ironic is that for someone aspiring to be a Lisper?)
(UPDATE: my recruiter was actually technical enough to say "the solution to the first problem was very elegant", and yet he didn't even look at the second solution because it didn't compile; he didn't care that it was almost finished, just that it didn't compile. So, once more: better get a bad but "working" solution than a good but unfinished one. Because "Worse is Better", right? ... how ironic is that for someone aspiring to be a Lisper?)
Finally, the recruiter gets Codility's automatic analysis on the code, which include correctness and performance checks, where of course you should get close to what was requested in the test's description. For correctness you can check with any small input vector (attention to corner cases, of course!), and the test description will provide you with one or two; but there is nothing for testing big input vectors, and being prepared to test with a 100K-element vector (random, or zeroes, or whatever) can be instructive. Again, having your own IDE/environment ready will help lots. In principle you should be already knowing how your code will behave with big inputs, but still I find that after 2 hours of stress seeing first-hand that a test goes through (when you can no longer even understand how some array index got off by 1!) is a relief.
Also, even though Codility's environment allows you to run the code and even prepare 5 different test inputs to run, it takes some seconds to launch a run. It's only seconds – but it can feel horribly long during a test, and can easily take you out of the zone.
Also, even though Codility's environment allows you to run the code and even prepare 5 different test inputs to run, it takes some seconds to launch a run. It's only seconds – but it can feel horribly long during a test, and can easily take you out of the zone.
And that's it. To me it's kind of a disturbing picture. The premise I liked about Codility was that the testing would let companies get good programmers, and programmers would get an indication about how good (or not) they are and which companies do care about it; but because of how the tests work in practice, it rather starts to sound like you have to train Codility-style to even take a look at the company / be checked by the company. To apply for an embedded-C job I have to train Python and psychic myself to ignore lots of details that make me a good C programmer - or lose to someone who did.
Worse, two years ago I already mentioned to them most of these problems. Things have barely changed; from what I remember, the only visible change for C is that now they tell you to use C99, when before it was unspecified and C99'isms just triggered warnings – which they didn't care about, but which as a C programmer made me change my program on test-time to get rid of them! So, doesn't really look like things are improving.
Of course, all of this is moot if you can get a perfect 100% score in the allotted time. I'm not there yet, to be sure – and I have the impression that the more I learn and use C, the worse my result is in this kind of test. Though my Python results improve :P. And that anyway sounds like just promoting competition programmers, which sounds uncomfortably narrow.
I wonder how things look for other languages. And how recruiters react to Codility's analyses. One hopes they'd wisely just treat it as a (useful) tool that only shows part of the picture. And yet, you have companies using things like the Taleo websites (shudder), so… maybe better not to get too hopeful.
I got codility test today. Thanks for sharing your experience!!! Its helping me alot.
ReplyDeleteThis comment has been removed by the author.
DeleteI would add a tip, Run the Solution(xyz) first with a print_r or other print statement to understand the input parameters better, I wrote a solution based on the example description of a tree structure that was written (1,1(3,2),(1,2,4),etc) and when I pasted and ran the code at the deadline, it failed because the input was an array. Stupid me.
ReplyDeleteThank you for your description of your experience with the Codility evaluation. I had figured that pulling my own IDE was the way to go. I am worried about corner cases getting in the way of reasonable score and of course recruiters fixating only on that single number.
ReplyDelete"God, grant me the serenity to accept the things I cannot change..." etc. you know the rest. Yeah, f-ck 'em if they can't see past your Codility scores. It's a sign from on high that you didn't need to work for the f-ckers anyway. ;)
DeleteYeah, just took a Codility test just now. I tried some of the Codility exercises first to warm up. Then took the test. Def. took me by surprise in 2 of the 3 exercises. I got a 10 on the second task but an overall score of 48. The recruiter said that if you get between 40 and 60 the team there reviews the code.
ReplyDeleteI do have to wonder how much laughing and carrying on must happen at those reviews. Like guys in a fraternity laughing about their time as pledges as they discuss how the current crop of aspirants did during some pledge activity.
It'd seriously freaking help if we knew what they were testing for... in Real Life, progs. know the metrics they are coding against so why not here?
I suspect Codility is an evil, vicious practical joke being played on the programming world by testers who are trying to get back at us for making their lives miserable.
I recently took a codelity test. I gave up because each of the tests were based on someone that knows the theory from a CS standpoint. I was never a CS graduate. Needless to say. I am tired of the trolling. I am going to learn the algorithms just so I can tell them after I take the test and ace it, to fuck off.
DeleteI will make the test in a day or 2 and while looking around for info found your site.
ReplyDeleteMy input is that the usual time a recruiter spends with a CV is less than 1 minute, the probability that a recruiter will look into your test behind the results and check the code is not null ... but very close to it!
Yep, and that's why it probably pays to quickly whip up any naïve-but-working solution, instead of going for a "better" solution that might not be finished on time.
DeleteWhich for example favors devs who don't know of anything better than a bubble sort vs those who know that the bubble sort is not a good solution for a big problem but can't implement any other sort under pressure.