Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts


YOU could have invented the LMAX Disruptor... if only you were limited enough!

This is about a shocking realization I got while learning about the LMAX Disruptor – a data structure / architecture that got published in 2011 and made some waves across the web because it allowed LMAX to process +6M transactions per second on a single thread. In Java. Which was interesting because Java was (still?) supposed to be too slow for that kind of thing.

By then I was already a couple of years out of the Java world and getting into embedded C, so I took a mental note to look into it just out of curiosity. But, as could be expected, the note was promptly forgotten.

I'm now looking again into Java, so finally I managed to take a peek at the Disruptor. Shock: this is just a description of what I was implementing in C while I was still learning it! But it's not that I'm a genius; it's that this is probably the only way to do things when in a limited environment. In fact, the basic architecture is the one used typically in drivers and in Linux's NAPI network architecture, which is in place since the early 00's. I see a handful of others across the web having commented as much during these years.

So... is THIS what merited such coverage, talks in conferences, etc?

I used to think that a surprising part of the embedded people I've met are typically rather short-sighted and closed minded about their part of the stack; now it's looking like the higher levels are also equivalently blindsided. So it sounds like there really is a need for some condensing view, some whole-picture consideration in the industry. But, how? Who?

I guess I could take this as a job opportunity :P, but it rather sounds depressing / worrying. If computing is akin to thinking... what does it say if we programmers seemingly are such poor thinkers? And I think that "specialization" would be a really poor excuse here. "Specialization is for insects" – right? Sounds to me more like oblivious churning out. Which might be well and good when making another templated website, but not as good when writing software to control, say, medical equipment.

So let's try to analyze how and why the Disruptor is or isn't interesting, and the implications.


User-site-installed python packages, and PATH modification

Python makes it apparently easy to install packages. Just use pip, or any of the other more-or-less old and deprecated ways to install them, right? (heh)

The first difficulty is that maybe your system's Python needs sudo to install those packages, and you don't want (or even can't) use it.

The definitive solution is to use virtual environments, but that can feel like going too far in the "local" direction. You might just want to have something at the user level, without having the risks of using sudo, but still global for everything that the user does.

Well, turns out that PEP 370 allows you to have user-local installs of packages. You're supposed to run pip --user install whatever.

But now you have to remember to always use the --user flag! Kinda breaks the purpose of having something "global for everything that the user does".


Grand Central Dispatch for Android?

(Sometimes after a couple of hours of research, the result is a dead-end or needs further digesting.
Still, such a result might itself still be worth remembering.
This is a Quick&Dirty report of one of those cases.)

GCD is an implementation of thread pools / queueing (+ event management (sockets, files), etc) by Apple for Mac OS X / iOS. It was open-sourced, and FreeBSD got an implementation. They are practically an abstraction and simplification over pthreads (+ events, etc), and the GCD implementation includes some kernel and compiler changes (workqueues and blocks, respectively). Implementation details at It's well performing and seemingly can help simplifying a lot.

What if one wanted to use GCD in Android? Is there anything compatible enough and that could be used at the C level?


GitFlow (and friends) with remotes: avoid the busywork

If your local feature branch tracks origin/feature, you are probably working extra, no matter your flow.

There are lots of places online where one can learn about GitFlow, but it's seemingly always discussed in a local way; the details, subtle or not, about using GitFlow when you are pushing and pulling your changes through the network are never mentioned.

Now, GitFlow is a bit long on the teeth, and some of the younger and simpler alternatives do take into consideration things as Pull (or Merge) Requests; lately some even want the original GitFlow to be considered harmful. But still, there's a common basic concept in GitFlow and the alternatives, which is local feature branches. How and when exactly to merge them back into a development branch is one of the big differences, but is rarely detailed.

The goal of this post is to gather some tips on how to keep developing in the local feature branch while staying involved on a lively repository; and how to make easier the final feature merging.


Doxygen error parsing in Eclipse CDT

Eclipse (or is it the CDT?) has, by default, some Doxygen integration: if the preferences are set appropriately, Doxygen-style comments are highlighted differently, and Doxygen commands in those comments are further highlighted.

But that's about it. A particular missing feature is that errors in the comments are not highlighted in any way; and that's what made me look for a better alternative.


repo sync --force-sync overwrites your existing repository!

I couldn't find any good explanation of what does Android's repo's --force-sync do, nor why it can be needed. So I'm reporting my painful findings in repo 1.22.


An IS_DEFINED() C macro to check whether another macro is defined

Every C programmer knows to use #defines and #ifdefs to disable compilation for a piece of code. But not everyone knows that there are better ways; the problem with this strategy is that the code becomes invisible to the compiler (and "smart" indexers), which means that it stops being checked for correctness. Also, if the rest of the code changes substantially, like variable renaming, function changing their signatures, etc, the disabled part falls out of sync and will need possibly serious work when re-enabled.

There are easy improvements to the #ifdef strategy, with the inconvenience that they add boilerplate and/or waste cycles on runtime (important on embedded environments). But there's also a way to remove both the boilerplate AND the runtime checks!


Tweaking Waf for integration with Eclipse CDT

First, the way not taken: there is an official Waf extra named "" which theoretically creates the files that Eclipse expects in a project. But that extra is from 2011; both Waf and Eclipse have evolved since then, and I didn't have the time nor inclination to dig into both Waf and Eclipse internals. Also, what would be the end result of that extra: only build support, or full indexing? No docs, no idea, forget it.

So let's make it easier. We already know that Eclipse's indexing system only needs to see the compiler command lines to get an idea of what files are related, where are the includes, what are the compiler flags, etc. So what about just making Waf spit out the command lines? And turns out, it's even easy: there is another extra called "print_commands" which does just that.


Flame Graphs vs Instruments in OS X: using Intel's Performance Counters

TL;DR: as of last update for OS X 10.11 / 7.1, you can't really create meaningful CPI Flame Graphs on OS X because of bugs in Apple's tools, both GUI and command-line. There are some alternatives and workarounds; for an alternative to a CPI Flame Graph, might be good enough. I sent 6 bug reports to Apple, and 5 of them got marked as duplicates; so maybe some Instruments post-7.1 will get good enough to allow the data extraction needed for real Flame Graphs.

I have been trying to generate Brendan Gregg's Cycles-Per-Instruction Flame Graphs on OS X (10.10), mostly to check cache-related CPU stalls. However, after working on it for a good while, it's looking to me like the given method is somewhat shaky, and the same or better insights about the code can be gotten more easily; partly thanks to, … and partly in spite of


List the DTrace providers in your machine

I don't see any official way to list the DTrace providers; you can seemingly only list ALL the probes, the >300K of them (in my Mac right now), and then you have to deal with the multitude of providers instantiated multiple times for different PIDs.

So here's a small AWK script to list the unique providers, how many instances of each are there, and how many providers are attached to each PID:


Best practices for a (GNU) makefile in the XXI century

This is not really a tutorial on makefiles; there are lots of those around the web.

But most of those are very outdated, and/or follow dubious practices. You'll end up with a makefile which was OK for a make and a compiler from the 90's, maybe even around year 2000. Even the GNU make manual recommends unnecessarily complicated things, given the capabilities gcc has grown in the last decade.

So, if you know how makefiles generally work (it's enough to know what a rule, a recipe and a variable is), and want to improve yours or know of better options, the following may help.


Fred Brooks vs Dijkstra?

In the 60's, Freed Brooks led the development of the IBM/360 system and its OS. The experience made him write the famous The Mythical Man-Month.

Turns out that Dijkstra also wrote about that system in his EWD255. And everything he complained about on that system is the standard nowadays! Sounds like he considered that a lot of the complexity that should be fixed by the computer is being dumped on the programmer; that the features touted by the IBM/360’s OS are in fact partial fixes to the bad design of the computer:


My first official Linux kernel patch

Well, that took some time. Quite an exacting process… which makes you get a new perspective of how Linux is made, and what it takes to keep loose collaboration at a high quality level.

And interesting too how such a simple patch evolved. First it made me think of the bike shed

git mental model

Finally I managed to get a mental model of git that allows me to do everything without needing to google around. And this is a quick memory dump.

There are other git cheatsheets, but this is mine.


Modelo mental de git

Por fin conseguí hacerme un modelo mental de git que me permite hacer de todo sin necesidad de google ni esquemas ni Stack Overflow ;P. Y ésta es mi chuleta / "volcado de memoria". Parece que no hay muchas cosas así en español; a ver si le sirve a alguien.

La idea era mantenerlo muy corto y al grano — como una chuleta, vaya. Asumo que sabes de qué va en general git y qué son cosas como un puntero y una lista simplemente enlazada. Si hay interés lo expandiré en algo más "tutorial".


Swapping bottom left and right panes (content and patch summary/file tree) in gitk

Gitk (as of git 2.4.2) has its bottom left and right panes the other way round to what I like: the bottom right shows the file tree or the patch summary, while the bottom left shows the contents of what is selected on the right. Isn't that kinda non-standard?

There is no official way to swap those panes, so here's a little hacky patch to do it.


Great minds think alike…

… or, "using quotes by well-known people to put forward possibly unpopular points" ;P

Sometimes I have a fully formed opinion to write about; lots of times I don't, but I have an intuition of where I want to go. And those times seeing what some of the "greats" in the field think can be a bit of a beacon.


Eclipse CDT configuration for big, makefile-based projects

It's been kinda hard to get Eclipse to work well with big CDT projects (say, the Linux kernel, or even a BuildRoot project including the Linux kernel). The documentation for Eclipse is between non-existent and unhelpful, and I only could find a post that talked specifically about preparing an Eclipse project for the Linux kernel. But that post explains nothing, and does some arbitrary things. So this is the generalization of that.

Telnet autologin with a short Expect script

Couldn't find anything concise, so here's my solution: a sweet short Expect (the CLI variety installed by default in Ubuntu) script to open a telnet session, feed in the user and password, AND continue to an interactive session.


Type punning, aliasing, unions, strict-aliasing, oh my!

Imagine that you have a struct s which sometimes needs to be volatile (because it maps to a set of hardware registers) but other times you'd rather have it non-volatile (because you are working on a bunch of such structs that have been stored, so they are no longer used to interface to the hardware and you could do without the performance penalty of volatile).

That is easy enough: you can have your variables declared volatile or not depending on the situation.

But what happens when you have a function to deal with those structs? Could it be implemented so it not only works, but does the right thing in both volatile and non-volatile structs?

The trivial way to have such a thing is by just defining 2 versions of the function (with different names of course) with the differently qualified parameters; but another possibility is to define a union type in which one member is the plain type and the other is the volatile-qualified type, and make the function parameters use that union type. Standard-wise, it should work… shouldn't it?