Showing posts with label networking. Show all posts
Showing posts with label networking. Show all posts


Telnet autologin with a short Expect script

Couldn't find anything concise, so here's my solution: a sweet short Expect (the CLI variety installed by default in Ubuntu) script to open a telnet session, feed in the user and password, AND continue to an interactive session.


Using OS X’s syslogd to receive log messages from the network

TL;DR: avoid this buggy mess and go with macports & syslog-ng. You'll finish faster and saner.

[Updated 2 times]

This sounds like should be easy, but OS X is a moving target because of all the infrastructure changes they have been making for the last few OS releases. Yes, there is a syslogd, but it is some half-hollowed out thing and “others” do most of the work. Syslogd does NOT open an UDP socket, launchd does and feeds it to syslogd. Syslogd does NOT (really) receive the UDP packets, a plugin does it. Syslog does NOT parse the UDP message, ASL (Apple System Log?) does. Syslogd does NOT filter the messages and store them into “logs”, ASL does.

So why there is a syslogd at all, apart from giving a slight sense of false security? (As in, “c’mon, there’s syslog, can’t be too difficult”). No idea. If I had seen how complicated this was going to get I would have bailed out and used syslog-ng from macports.

Anyway. So the first step is to enable UDP reception. The manpage for syslogd explains what has to be changed and where, but doesn’t mention the intermediate step of converting the syslogd plist into xml. Which is easy with plutil or by using some GUI editor, but the fastest way is to just change things from the command line:
cd /System/Library/LaunchDaemons
sudo /usr/libexec/PlistBuddy -c "add :Sockets:NetworkListener dict"
sudo /usr/libexec/PlistBuddy -c "add :Sockets:NetworkListener:SockServiceName string syslog"
sudo /usr/libexec/PlistBuddy -c "add :Sockets:NetworkListener:SockType string dgram"
sudo launchctl unload
sudo launchctl load

With that, log messages originating from the network will be dumped into the normal ASL message storage. Some of that goes into BSD-style, flat text files. But another part of it does not, and stays in some internal ASL database. So the best way to see everything kind-of-at-once is to use the OS X Console utility.

However, the UDP syslog protocol is supposed to include in the UDP packets some information that is not appearing in my Console. Most importantly, there should be a Host field, which would be perfect to classify the log messages coming from the network; alas, in my usage I am not seeing that. After checking the debug info from syslog and ASL and their dogs, looks like either the Linux side (DD-WRT) is not sending the packets correctly or the Mac side is not parsing them right (no, I didn’t feel like trying Wireshark), and some fields are either swapped or missing-and-not-accounted-for, causing some fields to contain values that should be in other fields. Which results in the log messages coming from the network getting lost among the local log messages… and there is no direct way to distinguish them.

So, next idea: dump the log messages coming from the network into a separate file.
To do this, it’s necessary to add a rule to ASL so it filters out the messages that came from the network and writes them into our file.
The problem is that once the messages are inside ASL, already there is no direct way to know where they came from! Luckily, they can still be indirectly recognized because of some anomalies: for example their GID, UID and PID are nonsensical and always the same: GID = UID = -2, PID = -1.
(note that the GID, UID, PID and other keys are not usually in the BSD-flat-format typical in logs; but ASL does store metadata for every message, and a lot of that metadata is simply ignored when outputing to the BSD format. So the idea of writing the UDP-received messages into a file is not the best solution, and I would have preferred being able to continue using some ASL storage and to comb through it; alas, I couldn’t find a way)

After some tests, this is what worked for me: put the following into the file /etc/asl/syslog_UDP :
> my_syslog.log mode=0740 format=std rotate=seq compress file_max=5M all_max=50M gid=20
? [N< PID 0] file my_syslog.log

And signaled syslogd to reconfigure:
sudo killall -SIGHUP syslogd

Aaaand it’s working! In /var/log a folder named syslog_UDP will appear, in which logfiles named my_syslog.log will contain log messages with a PID lesser than 0, which should be impossible - but that is what the UDP-originating messages report. Note too that the logfiles will be automatically rotated, compressed and purged after they start taking more than 50 MB. And they are readable by everyone - no sudo needed. (You are not running as an admin, are you?)

This will break when either DD-WRT starts sending correct UDP messages, or when OS X starts parsing them right, or when the syslog/ASL mechanism changes…

One thing that I tried and couldn’t get to work is using “directory my_syslog.log” instead of "file...", but the result seems to be buggy, in that the generated files inside the directory do not honor the mode or UID/GID that I try to set.
And the debug support in ASL (activated by adding a line “= debug 1” in /etc/asl.conf) is less than useful. Finally what helped the most was to use the message inspector in and the format=raw output flag to see the key values as they are at filtering time - the -1 and -2 values are in fact shown as the decimal value of 0xFFFF and 0xFFFF+1 in the message inspector, which did not work when filtering

In hindsight I’ll bet that using syslog-ng, or even some hackish reverse netcat, would have been much faster. Buuut… well, at least I have learnt a bit about the logging madness in OS X, which in fact had been kind of a to-do for some years…


UPDATE 3/3/2015: Beware, this seems to stop working after a while. My first troubleshooting shows OS X's syslogd to be buggy when receiving from UDP (in Yosemite at the very least): syslogd stopped working (seemingly at all, not only the UDP reception) after some hours, and lsof shows that syslogd had 255 file descriptors open, most of them to the UDP socket. Sending a SIGHUP didn't do anything. "sudo killall syslogd" restarted it and now seems to be working, but who knows for how long. So I see myself going the syslog-ng/macports route in the near future.

UPDATE 4/3/2015: the UDP file descriptor growth keeps happening. Seems to be related to my WiFi connection going on and off. Nastily unreliable, and a funny way to remotely disable syslog on an OS X system, too. I'll disable the UDP thing and send a bug report. My recommendation: just go with macports & syslog-ng.


Selecting a mobile internet operator in Poland

(see the 2014 update)

About one year ago I was tired of the problems with my then-current internet provider, Play. So I decided to try the 4 major operators in Poland. I got a "starter pack" for each one, which can be bought for about 10 zl each, and tried them with my telephones/modems/computers. These are the results of testing with,, youtube's speed test and torrenting some big Linux disc image. And since this is mobile internet, I tried each in a set of places in Warsaw - and some even outside of the city.

Note that I was looking for smartphone use: voice + data, with data being in the 5 GB/month range, possibly more. So data packages measured in MB didn't cut it (since they are usually much more expensive). And internet-only SIMs were out, too.

The tests are one year old, but I can confirm that the results are still valid at least for T-Mobile and Play, since I still use them both.

For non-technically inclined people, the summary is: get T-Mobile.  Play is unreliable, but cheapest. Orange is surprisingly crappy. And Plus didn't even have anything serious for smartphones (that is, for voice + data "in big quantities").

And the gory details:

Era / T-Mobile: the best one. One easily reaches the advertised speeds, while testing and downloading. Pingtest gives marks between 4.0 and 4.2 (C, sometimes even B). Pings between 160 and 190 (!), Packet Loss 0, jitter typically in the single digit (!), sometimes up to 20. Youtube averages 19.
It's not even the most expensive of the operators, if I remember correctly. The problem is that once you use up your monthly allowance, you can NOT extend it. So if you need more than 5 GB, you're toast. Their internet offer is advertised as unlimited, but that only means that after you use up your paid package, you can continue using internet at about 2 or 3 KB/sec. Practically unusable if you can't limit yourself to text.
Era is my main operator, so I can confirm that it still works well - even if I had some complaints at some moments because of weird connectivity problems, which were quickly and personally answered - even if in a useless way. :P

Play: the cheapest internet operator. Too good to be true, certainly. Half the price of T-Mobile, can be refilled as you wish, there are even options to use internet during the night without it counting against your allowance. But, even before getting into the numbers, it is plainly shitty. On average, more than once a day the telephone or modem will loose the connection; sometimes in "standard" ways ("the PPP server has stopped responding"), but more frequently the phone just suddenly shows no bars of network reception. Both the telephone/modem and the computer wait for it to come back, which never does - I have tried waiting for hours. You can just restart the phone and it will connect again, but sometimes in less than a minute it will happen again. I started thinking it was some problem with my telephone, with drivers, with the mac... But, it has happened with different computers (although always macs), with different telephones and modems, even using a VM with Windows to drive the modem. Same thing always. So, I can NOT recommend Play at all, for any serious internet usage at least. No idea about how it works for voice. And maybe for short connections, like those typical in a smartphone would be usable. I wouldn't risk it, though.
Anyway, I also tested it "numerically", and the results were: pingtest marks between 1.6 and 3.9 (F to D). Pings between 140-250, jitter between 80-260, packet loss 0. The max standard speed is 125 KBps. There is an option to get 400 KBps, but I feel lucky if I get just half of that.

I STILL use Play for "bulk internet"; it is so cheap (even more so with the free nights option) that it always gets me to try again. But it also always manages to get on my nerves; 3 times already in 4 years it has made me prefer to pay extra to other operators... and the 4th is coming soon. SO annoying, SO unreliable, SUCH helplessly blank faces in the hip shops when I tried to get some info. Fuck them.
Also: Play only provides HDSPA connections. If at any moment the phone has to fall back to EDGE, you're as good as with no coverage. That was the case when I tried it at my workplace, which is only about 2 km away from the city center: I couldn't even connect.

Orange: the only operator which showed packet loss (!) in pingtest. Marks between 1.1 and 3.9. Pings up to 440. Jitter between 18 and 205. I never reached the advertised speeds for downloads, but did for uploads. Keep in mind that I tried in different locations, with different modems and telephones, so I can only think that this was simply Orange being just shitty.

Plus: this was my first operator in Poland, and maybe that's why I had something of a soft spot for them. And it was impressive to be able to work with SSH and VNCs almost all the way in the train between Warsaw and Tricity. With Play, you'd loose coverage before you leave the city!
Alas, when I tested them a year ago they didn't have any specially competitive offer. In fact the only data+voice / smartphone-related offer they had was something tied to BlackBerries (heh, I wonder how are they doing now with that?). It was funny that the shopkeeper seemed rather desperated about the smartphone questions; it felt like I was n-th person to ask the same thing... and to be surprised that they had no options.
Anyway, I also tested with pingtest. Marks between 1 (!) and 4.25 (!). Pings between 134 and 388. Jitter between 13 and 104. No packet loss. Easy to reach advertised speed.

So, when looking for reliability and sanity keeping, T-Mobile seems the only one commendable. Also, with that small jitter, Skype and such were great. People prefer videoconferencing when I'm in the phone via T-Mobile than when I am in the computer via Play...

What I can't understand in T-Mobile is the lack of "refills": I just want to give them more money, after all :P. But, no way. So stupid.

A few more details: when testing with, I used the auto-selection of test sites. Each operator seemed to consistenly cause a different test site selection, so I guess it was the best one for each of them, surely according to ping times.

It's remarkable that T-Mobile has such a low jitter (and consistently so), that Orange reliably suffered packet loss (and was the only to suffer it at all), that Play is so damn unreliable, and that Plus... well... HELLOOO!! wake up and smell the 21.1 century!


Safari's Reading List protocol: CalDAV + XBELs

Safari on desktop and iOS has a feature called "Reading List". It is a way to store URLs in iCloud, synchronize them between Safaris, and mark them as read or not. Somewhat like Instapaper maybe.

I was a bit surprised that there is no Chrome or Firefox extensions for tapping into Safari's Reading List. So I wanted to try poking a bit into the protocol, maybe something interesting would appear or someone could get some headstart from this.

Safari always seem to start contacting, which resolves to the Akamai CDN, some Luckily the IP always was the same, and there is no client certificate check, so I could mount a little Man In The Middle via /etc/hosts.

Then, a couple of socats: one to pose as the original SSL server and resend the plaintext; and the second to receive the plaintext and send it to the original IP. The first socat shows everything that goes out of it (in plaintext) thanks to the -vx option.

socat -vx OPENSSL-LISTEN:443,cert=certs/server.pem,verify=0,reuseaddr,fork TCP:localhost:50000 

socat -v TCP-LISTEN:50000,reuseaddr,fork OPENSSL:

I tried tcpflow instead of the -vx, but tcpflow won't show anything when capturing on the loopback interface on OS X. Seems to be a bug.

So now we can see the exchanges between Safari and iCloud. I was half-expecting something encrypted, but it isn't (apart from the SSL connection). It seems to be CalDAV, used to exchange small XBEL files, gzipped; which can be quickly unzipped by copying the hex from the socat dump and pasting into a "xxd -r -p | zcat ". Each XBEL file is a small XML file with an URL and its status.

I expected this to be much quicker, but socat and tcpflow kind of conspired against it. I started expecting to use named pipes and tee's to connect the socats while showing the exchanges, but each update in the Reading List causes between 3 and 5 connections, which needs the socat option fork, which forks new socats for each connection, which won't play nice with the pipes.

I guess I should have switched to some python earlier... next time I will. And at this point anyway, if I wanted to keep going into this protocol, looks like throwing together some small client with Python would be the best way forward.

But I am not currently interested in JavaScript, so I guess I won't be making any extension myself, so I guess I should move on to other things.

So, dear Lazyweb... :)

Huawei E220 3G modem drivers on OS X Lion: only 32 bits

I recently installed Mac OS X 10.7 Lion, and found that it already had drivers for the Huawei E220 3G modem. But the drivers are 32 bit, so they won't work for machines which use a 64 bit kernel.

And there seem to be no 64 bits drivers, so the only solution for now is booting in 32 bit kernel mode (pressing 3 and 2 when booting).

I seriously doubt that Huawei are going to publish updated drivers; they were already rather unsupporting even when the modems were new. Eventually, I plan to try to make my own driver, but that won't be short-term...


Tcpflow and connections between local interfaces

Looks like tcpflow doesn't see connections between local interfaces. After a bit of digging, looks like such connections are "routed internally by the kernel", at least in Linux. And there are patches for Linux to force those packets out of one interface and in from another, but even that is only useful if you have an external network connecting both interfaces (looks like a simple crossover cable should be enough).

There is another option: using iptables to first make the packets leave the machine towards some external IP, and then using arpd to make a router send back those packets.

And I see people reporting that tcpflow -i lo does work for them, capturing flows having local addresses even though different than

The interesting thing is that people seem to take for well-known that Linux routes through the "lo" interface the traffic between local interfaces; but I didn't find any authoritative source which explains any rationale, any configurability, any implementation. Which I guess would make it somewhat easier to find such things in the BSD's and OS X.

(I surely should go straight to the source code, but that feels fragile. I am not interested on the current implementation, but on the design: how should it work vs. how does it work right now. Although surely that kind of networking would be pretty baked in into the kernel...)

Would be interesting to know if this something missing in the BSD's/OS X or in libpcap.


tcpflow 1.0.2 doesn't work with net expressions

Tcpflow 1.0.2, as built with MacPorts, doesn't work when net expressions are used.
But 1.0.6 does work.

I already submitted a new portfile so it should be available soon.