This post is meant as a reference point–to illustrate that, according to a wide range of accounts, partitions occur in many real-world environments. Processes, servers, NICs, switches, local and wide area networks can all fail, and the resulting economic consequences are real. Network outages can suddenly arise in systems that are stable for months at a time, during routine upgrades, or as a result of emergency maintenance. The consequences of these outages range from increased latency and temporary unavailability to inconsistency, corruption, and data loss. Split-brain is not an academic concern: it happens to all kinds of systems–sometimes for days on end. Partitions deserve serious consideration.

I was running through The Advanced Bash-Scripting Guide's "Complex Functions and Function Complexities" (a long but interesting read), and was struggling with some unexpected behaviour - a variable in my the while block of a script wasn't incrementing as expected.

curl "$THIS_URL" | sed -e "s#foo.push(##g" -e "s#}});#}}#g" > $TMP_FILE
while read line; do
  ...
done < $TMP_FILE

I found the answer here.

It turns out that this is because pipes cause Bash to perform operations within sub-shells, and variables are specific to whatever sub-shell they're called in. So, once we leave the do/while loop, $COUNTER will again evaluate to 0. Lame!

And refactored my script to use process substitution instead of pipes like this.

while read LINE; do
  ...
done < <(curl "$this_url"="" |="" sed="" -e="" "s#foo.push(##g"="" "s#}});#}}#g")="" <="" pre="">

Of note is that using this hashbang at the top of the script

#!/bin/sh
 

failed to execute the process substitution, but replacing it with the binary it was symlinked to (/bin/bash) worked just fine.

Jun 25 10:59:13 localhost postfix/local[20793]: fatal: open database /etc/aliases.db: No such file or directory

As the missing file is some form of (Berkley?) DB, a simple touch doesn't create any structure for a suitable "empty database".

To fix, update the /etc/aliases database with newaliases:

newaliases

From the man page:

This is a link to the ssmtp binary. It invokes /usr/sbin/sendmail with the -bi option. It is provided for compatibility with the sendmail program. In this case it does absolutely nothing since sSMTP does not support /etc/aliases and is just there to avoid programs returning error messages.

It can be infuriating when resizing a window, to have Amazon's AWS Dashboard minimise all columns. To restore them back to their sensible layout:

• Click Show/Hide
• Toggle any column
• Click Apply
• The columns will have reordered (and you can now un-toggle the added column)

node-inspector is the de-facto standard Node.js debugger, but runs in another browser window. For those that like their IDEs fully integrated, PhpStorm should support remote debugging functionality for Node.js but is currently buggy - it will only connect to servers on the same host. The bug is here.

For when the bug is fixed, here are RHEL/CentOS instructions on how to connect to a remote Node.js process (Node.js currrently only allows its debugger to listen for connections on 127.0.0.1, which isn't much use for remote debugging).

On VM console 1 (-df debug mode on for diagnosis):

yum install balance -yt
balance -df 8585 127.0.0.1.5858

On VM console 2:

node --debug app.js 

On host:

telnet {VM IP} 8585

Should output

Type: connect
V8-Version: 3.6.6.14
Protocol-Version: 1
Embedding-Host: node v0.6.6
Content-Length: 0

VM console 1:

$ balance -df 8585 127.0.0.1:5858
argv[0]=balance
bindhost=NULL
source port 8585
file /var/run/balance/balance.8585.0.0.0.0 already exists
the following channels are active:
  0  0 127.0.0.1:5858:0
connect from 172.27.4.11
trying group 0 channel 0 ... connect to channel 0 successful
connect from 172.27.4.11
trying group 0 channel 0 ... connect to channel 0 successful

The connection is working, so attach the remote debugger in PhpStorm (this is the step that's currently broken):

Debug on!

Google Calendar caches remote calendar URLs for some time - not helpful if you want to verify a change.

This can be circumvented by altering the URL while requesting the same content. Adding a query parameter (or attempting to add another one) will cause Google to treat the calendar as a new feed.

Here's an example with the UK holidays calendar feed from CalendarLabs:

Original Calendar URL:

webcal://www.calendarlabs.com/templates/ical/UK-Holidays.ics

Updated Calendar URL:

webcal://www.calendarlabs.com/templates/ical/UK-Holidays.ics?noCache

And another example with an updated query string from Facebook:

Original Calendar URL:

webcal://www.facebook.com/ical/u.php?uid=123456789&key=ABCdefGHIjkl_m123

Updated Calendar URL:

webcal://www.facebook.com/ical/u.php?uid=123456789&key=ABCdefGHIjkl_m123&noCache

Any string can be used instead of noCache, as Google Calendar just remembers the full URL. So next time you want to force an update, use noCache-2, noCache-3 etc.

If the calendar's URL already has query parameters, or adding another parameter breaks the feed, the URL can be shortened with bit.ly (or any URL shortener). This is less useful, as you're relying on the query shortening service always being available.

SSH connections on Windows are best managed through PuTTY, although the client only allows one connection per window. For marshalling many PuTTY instances, PuTTY Connection Manager is my preferred option (other window managers are available, including superputty).

Unfortunately the URL of the PuTTY Connection Manager website is returning 403 Forbidden, and there don't appear to be any mirrors of the last build that was released.

Archive.org doesn't have the download page archived, so there are no checksums available to verify my own archive. I've hosted the most stable version here for posterity: Putty Connection Manager 0.7.1 Alpha

If you already have this software installed, it is no longer able to check for updates from the target domain. If "automatically check for updates" is on and PuTTY Connection Manager crashes at startup, set this registry value to 0:

HKEY_CURRENT_USER\Software\ACS\PuTTY Connection Manager

Keeping the system clock on the right time is troublesome for a virtual machine if VM guest tools are not available to synchronise clocks. As the guest OS does not have access to the hardware clock it relies on software to tick over its own clock, and inaccuracies generally lead to slowdown (although over-compensation for loss of timer ticks may gain time).

Changing any system's time can be risky - an OS relies on a contiguous series of time values. If a system event is scheduled to occur at a specified time and the clock is set forward beyond it, that time never occurs and the event will not occur.

Network Time Protocol (ntpd) avoids this problem by making minuscule adjustments to the clock frequency (the slew), effectively shortening the length of a unit of time to the OS. man ntpd states:

The slew rate of typical Unix kernels is limited to 0.5 ms/s, so for every second that the clock requires adjusting, 2000s are required to adjust it.

For a development VM that is off for periods of time this slew rate is too slow, and if the machine doesn't run time-critical services ntpd can be configured to absolutely set the system time.

What

• Synchronise clock of virtual machine using ntp

Why

• Guest additions not available

Risks

• Time-based system events not occurring

How

1. Edit /etc/sysconfig/ntpd and add the -x flag (mine looks like this):

   OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -x"

   On startup the ntpd daemon will set the time irrespective of how incorrect it currently is.
2. Add this line in the root crontab to refresh the time every 5 minutes.

   */5 * * * * /sbin/service ntpd restart >> /tmp/ntpd.log 2>&1

Here is a detailed explanation of Timekeeping in VMWare and What is a Reference Clock? from ntp.org's FAQ.

A Kindle will sync your progress through a book when it's powered down (presumably only if it has sufficient battery to connect to a mobile or Wi-Fi network). This allows another device (another Kindle or the software on a mobile device or PC) to download that location so you can resume your reading from the same point.

If this doesn't appear to be happening from the physical Kindle (i.e. your progress isn't retrieved when you use a different device), the most likely reason is a corrupt filesystem. To fix:

1. Attach Kindle to a PC via USB
2. Your PC may prompt Do you want to scan and fix Kindle? - if so, click Yes
3. If your device hasn't prompted you:
   1. Navigate to Windows Explorer
   2. Right click on your Kindle
   3. Select Properties
   4. Navigate to the Tools tab
   5. Under Error-checking click "Check now"
   6. Select "Automatically fix filesystem errors" and click "Start"



4. Once the filesystem is repaired the Kindle will sync to Whispernet again.
5. In future, unmount the Kindle by ejecting it safely or change its removal policy.

I received an HTTP 500 response from YouTube today after submitting a search for Shlomo (an excellent beatboxer and human voice percussionist) and received the cryptic error mesage below.

It looks like potentially sensitive debug data have been serialised and encrypted into the error page, but it's not immediately obvious why this is fed back to the user and not logged internally. Perhaps this error:

1. occurred in an isolated system component without write permission to persistent storage
2. is a trigger condition for a reroute from an overloaded component (possibly an aggregation service?)
3. is infrequent enough to be considered non-critical
4. is an undesired, undiagnosed emergent property of their architecture
5. is unexpected, consequently not routed to the correct engineer and mothballed

Whatever it is, there isn't any direction on how to contact the "highly trained monkeys" so most users won't bother. The error is either considered minor or unfixable.

Serving 2 billion videos a day is non-trivial, but for any site without such well-publicised architectural prowess giving users encrypted debug messages and no method to report them is considered bad form!

ERROR 1137 (HY000): Can't reopen table: 'tmp_journals'  

It transpires that since 4.1 the way MySQL handles temporary tables has changed. This affects joins, unions and subqueries. There is an obvious fix:

mysql> CREATE TEMPORARY TABLE tmp_journals_2 LIKE tmp_journals;  
Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO tmp_journals_2 SELECT * FROM tmp_journals;  
Query OK, 3228659 rows affected (2.01 sec)  
Records: 3228659  Duplicates: 0  Warnings: 0  

Then the query is easy:

SELECT COUNT(1) cnt, journal_invoice_ref  
FROM tmp_journals  
GROUP BY journal_date  
HAVING cnt > 10000

UNION

SELECT COUNT(1) cnt, journal_invoice_ref  
FROM tmp_journals_2  
GROUP BY journal_invoice_ref  
HAVING cnt < 10