TLDR

Problem: trying to connect to a remote server with mosh you see error messages like:

mosh-client needs a UTF-8 native locale to run.
mosh-server needs a UTF-8 native locale to run.
The locale requested by LC_CTYPE=UTF-8 isn't available here.
locale: Cannot set LC_CTYPE to default locale: No such file or directory

OR when connecting with ssh from iTerm2 you get the error message

-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

Solution: If you are not in US/GB/CA/IE/AU/NZ, add the following line to your .profile and restart the shell.

export LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8

I am quite a user of mosh – a mobile shell that makes it really convenient to work via ssh over unstable Internet connection or when roaming across different wifi networks. It’s one of the first tools I install on a new server and from then on I switch from ssh to mosh.

This weekend I was doing an initial configuration to a new server I provisioned from OVH hosting. To my surprise, mosh failed to connect with the following error message:

$ mosh myserver.at.ovh

mosh-client needs a UTF-8 native locale to run.

Unfortunately, the client's environment ([no charset variables]) specifies
the character set "US-ASCII".

LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=
$ 

This was quite a surprise as surely the terminal in MacOS has a UTF-8 native locale. I changed to iTerm2 and tried connecting again, and again ended up with the similar looking error:

$ mosh myserver.at.ovh
/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
The locale requested by LC_CTYPE=UTF-8 isn't available here.
Running `locale-gen UTF-8' may be necessary.

The locale requested by LC_CTYPE=UTF-8 isn't available here.
Running `locale-gen UTF-8' may be necessary.

mosh-server needs a UTF-8 native locale to run.

Unfortunately, the local environment (LC_CTYPE=UTF-8) specifies
the character set "US-ASCII",

The client-supplied environment (LC_CTYPE=UTF-8) specifies
the character set "US-ASCII".

locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=
LC_CTYPE=UTF-8
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
Connection to myserver.at.ovh closed.
/usr/local/bin/mosh: Did not find mosh server startup message. (Have you installed mosh on your server?)
$

Apparently, something was wrong my with locale. Here I realized I have recently done a complete clean re-install and never restored whatever locale settings I had in my .bash_profile.

This actually only increased my curiosity: a freshly installed MacOS with still almost default configuration was causing mosh to fail on connection over locale settings. I googled a bit the mosh error message but didn’t find anything illuminating; then I noticed that when I was connecting to the server with ssh, iTerm2 would emit the following error message:

$ ssh myserver.at.ovh
Last login: Mon Oct  5 11:00:30 2020 from 178.136.74.184
-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

Interestingly, this message wasn’t showing up if I was running ssh from Terminal. So I googled this message in connection with iterm2 and found the issue #5478 with the comment showing an excerpt from iterm2 debug log:

getLocale: languageCode=en, countryCode=ZA
Tentative locale is en_ZA.UTF-8
Locale is NOT supported
Set LC_CTYPE=UTF-8

Now this was illuminating! iTerm2 calculates the value for LC_CTYPE variable by getting the language and country codes from the MacOS. When it ends up with a combination which MacOS lacks locale definition for, it falls back to the default LC_CTYPE=UTF-8. Terminal just defaults to LC_CTYPE=C which is non-UTF8!

Let’s see what English locales MacOS does have the support for:

$ cd /usr/share/locale/
$ ls -1d en_* | column -c 80
en_AU                   en_CA.UTF-8             en_NZ.ISO8859-1
en_AU.ISO8859-1         en_GB                   en_NZ.ISO8859-15
en_AU.ISO8859-15        en_GB.ISO8859-1         en_NZ.US-ASCII
en_AU.US-ASCII          en_GB.ISO8859-15        en_NZ.UTF-8
en_AU.UTF-8             en_GB.US-ASCII          en_US
en_CA                   en_GB.UTF-8             en_US.ISO8859-1
en_CA.ISO8859-1         en_IE                   en_US.ISO8859-15
en_CA.ISO8859-15        en_IE.UTF-8             en_US.US-ASCII
en_CA.US-ASCII          en_NZ                   en_US.UTF-8

Aha, not that many countries are supported, just the Five Eyes and Ireland! That explains why that person from South Africa ran into this issue with en_ZA and iTerm2: every combination of English as the OS language with an odd country will result in it. In my situation, the country was Ukraine with the country code UA, and of course there is no en_UA locale.

How does this affect the remote side of mosh connection – the server I am connecting to, though? Well, ssh – and mosh uses it for authentication phase – by default forwards the LANG and LC_* environment variables to the remote server it is connecting to:

$ grep -n -B1 SendEnv /etc/ssh/ssh_config
48-Host *
49:     SendEnv LANG LC_*

This makes the remote side of the connection declare them, and then mosh-server refuses to start because it requires a UTF-8 native locale to run. When I was connecting from Terminal, the locale was set to C which is not UTF-8, as I mentioned above; when I was connecting from iTerm2, the locale was defaulted to UTF-8 and this also irked mosh-server because there was no definition for it in the system.

How to fix it?

I could generate en_UA.UTF-8 locale definition locally, and that would make terminals define correctly LC_* variables, but they will send them over to the servers I am connecting to, and there mosh-server (and probably some other commands) will fail. This is no good.

Generating en_UA.UTF-8 locale definitions on all the servers I am connecting to is out of question too. Just like declaring overriding LANG/LC_* variables in .bash_profile files on all the servers: this might break legitimate clients with non en_UA.UTF-8 locales.

First working solution would be to disable LANG/LC_* variable forwarding in ssh configuration. Unfortunately, it is set in system-wide /etc/ssh/ssh_config and cannot be overridden in per-user ~/.ssh/config. I hate changing system-wide default setting with no good reason, and the problem with locale didn’t seem to me as a good reason to warrant such change.

Therefore I have opted for another option: just override LANG/LC_* variables in my user’s .bash_profile. This is a more klugy option, but in my situation it looks as the most reasonable thing to do. I have just added the following line to my ~/.bash_profile and it resolved the issue completely:

export LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8

I have been re-reading Tom Limoncelli’s et al. The Practice of Cloud System Administration book (which is great and is well worth reading even 5 years after it has been published) and it has this wonderful reference table called Latency Numbers Every Engineer/Programmer Should Know. This table was popularized by Jeff Dean, and originally presented by Peter Norvig.

I find it handy and wanted to copy it here, on my blog, but then realized it’s almost a decade old as the data was from 2012. This made me do a little research (read: googling) to see how the decade affected the numbers listed. The answer: not much! The only noticeable changes are in disk and network performance thanks to better SSDs/NVMe and 10/100Gb networks.

Which googling though I discovered two interesting bits though: some nice representations of data, and a scaled similar systems latency table from Brendan Gregg’s Systems Performance book. Besides of nice scaling of the latency numbers, that table neatly arranges different subsystems by speed: cpu, memory, disk io, networking, and reboot times.

I have added a scaling columns to the original latency table and updated numbers a bit in both tables to reflect current situations and putting them below for reference.

Latency Numbers Every Engineer Should Know

3GHz CPU cycle                     0.3 |   1 s    |
L1 cache reference                 0.5 |   2 s    |    
Branch mispredict                  3   |  10 s    |
L2 cache reference                 5   |  20 s    | ~10x L1 cache
Mutex lock/unlock                ~20   |   1 min  |
Main memory reference            100   |   5 min  | 20x L2 cache, 200x L1 cache
Compress 1K with Zippy         2,000   |   2 h    |
Send 1K over 1Gbps network    10,000   |  10 h    | 10Gbps is 10x faster, duh
Read 4K randomly from SSD    150,000   |   6 days | ~1GB/sec SSD
Read 1 MB seq from memory    250,000   |  10 days |
Round trip within same DC    500,000   |  20 days |
Read 1 MB seq from SSD*    1,000,000   |   1 mo   | ~1GB/sec SSD, 4x memory
Disk seek                 10,000,000   |  10 mo   | 20x DC roundtrip
Read 1 MB seq from disk   20,000,000   |   2 yrs  | 80x memory, 20x SSD
Send pkt CA->NL->CA      150,000,000   |  12 yrs  |
                        |   |   | ns|  | ~scaled  |
                        |   | us|
                        | ms|

I have updated disk/net numbers with data from this handy site which provides reference historical performance data over the last few decades.

Example Time Scale of System Latencies

3GHz CPU cycle                  0.3 ns  |   1 s    |
L1 cache access                 0.5 ns  |   2 s    |    
L2 cache access                 2.8 ns  |   9 s    |    
L3 cache access                12.9 ns  |  43 s    |    
Main memory access            120   ns  |   6 min  | 
Solid-state disk IO       150,000   ns  |   6 days |
Rotational disk IO             10   ms  |  12 mo   |
Internet: SF to NY             40   ms  |   4 yrs  |
Internet: SF to UK             80   ms  |   8 yrs  |
Internet: SF to AU            185   ms  |  19 yrs  |
TCP packet retransmit           3   s   | 317 yrs  |
Container OS reboot             4   s   | 423 yrs  |
SCSI command time-out          30   s   |  3k yrs  |
VM reboot                      40   s   |  4k yrs  |
Physical system reboot          5   min | 32k yrs  |

References:

• Peter Norvig’s original source
• Latency Comparison Numbers (~2012)
• Python: Real-Time Streaming Data
• Brendan Gregg’s table 1 and 2

Google Photos on iOS does not have a slideshow function! 😱 They did however roll out a new designed and killed off For You feature.

• The American state of New Jersey bans doctors from declaring someone dead on the basis of irreversible brain damage if they have reason to believe it would contravene the patient’s religious convictions.
• Why companies struggle with recalcitrant IT
  • Airlines are, for instance, now advised to turn the plane off and on again every 51 days, to stop its computers displaying false data in mid-flight. A similar problem found in 2017 in some aeroplanes made by Airbus, Boeing’s European rival, prompted the European Union Aviation Safety Agency to require that such aircraft be rebooted at least every 149 hours.
  • An industry rule of thumb is that, depending on how carefully they work, programmers make between 0.5 and 50 errors in every 1,000 lines of code they write.
  • According to Motor Authority a Ford GT has over 10 million lines of code, that is much more than what an aircraft needs to fly (2 million lines of code for the Lockheed F-22 Raptor and 7 million lines for the 787 Dreamliner).
  • At CES 2016, Ford indicated that they have 150 million lines of code in their new pickup, the F150
• Unix Wildcards gone wild – a list of unexpected behaviour you can run into when using wildcards

Targets

Targets create three common problems:

1. They produce perverse results when people focus excessively on them.
2. They tempt managers to manipulate numbers.
3. The obsession with measurement diverts people from useful activity to filling in forms.

The department of health provided a fine example of the first when it penalised hospitals whose emergency departments took too long to treat patients after ambulances had dropped them off. Hospitals responded by keeping patients waiting in ambulances rather than in emergency departments. The Metropolitan Police illustrated the second, after it linked pay and promotion to achieving a crime-reduction target. A police whistle-blower told a parliamentary committee that downgrading or underreporting crime had become “an ingrained part of police culture”. The universities to which a-level students are struggling to get admitted provide an example of the third. Tenure and promotion are awarded on the basis of the production of articles (which can be measured) rather than teaching (which can’t), so students suffer.

Tracking

Mr Netanyahu is instead banking on a controversial mobile-phone surveillance program created by Shin Bet, the security service. The program, designed to track terrorists, was used early in the outbreak, then halted. In June the prime minister brought it back over objections from the head of Shin Bet. The program is meant to identify those who have recently been in contact with an infected person. The health ministry then sends them a text message telling them to quarantine. But there has been a high number of false positives, say experts, and the public’s trust in the government is so low that many Israelis leave their mobile phones at home to avoid being tracked and ordered to quarantine.

This is just a quick note for how to make Home and End keys behave sensibly, i.e. take you to the beginning or end of a line:

mkdir ~/Library/KeyBindings && touch ~/Library/KeyBindings/DefaultKeyBinding.dict

Then paste into DefaultKeyBinding.dict the following content:

{
  "\UF729"  = moveToBeginningOfParagraph:; // home
  "\UF72B"  = moveToEndOfParagraph:; // end
  "$\UF729" = moveToBeginningOfParagraphAndModifySelection:; // shift-home
  "$\UF72B" = moveToEndOfParagraphAndModifySelection:; // shift-end
  "^\UF729" = moveToBeginningOfDocument:; // ctrl-home
  "^\UF72B" = moveToEndOfDocument:; // ctrl-end
  "^$\UF729" = moveToBeginningOfDocumentAndModifySelection:; // ctrl-shift-home
  "^$\UF72B" = moveToEndOfDocumentAndModifySelection:; // ctrl-shift-end
}

Some also like to remap ctrl- keybindings:

  "^\UF702"  = (moveWordLeft:); // ctrl-left 
  "^$\UF702" = (moveWordLeftAndModifySelection:); // ctrl-shift-left
  "^\UF703"  = (moveWordRight:); // ctrl-right
  "^$\UF703" = (moveWordRightAndModifySelection:); // ctrl-shift-right

You’ll need to restart MacOS for the change to take effect.

Sources:

• https://damieng.com/blog/2015/04/24/make-home-end-keys-behave-like-windows-on-mac-os-x
• https://coderwall.com/p/ikyhag/fixing-home-and-end-keys-and-more-on-macos
• http://cobus.io/osx/2017/02/09/OSX_Home_End_Keys.html

Wonderful Burns’ epigram in Marshak’s translation:

In se’enteen hunder’n forty-nine, 
The deil gat stuff to mak a swine, 
An’ coost it in a corner; 
But wilily he chang’d his plan, 
An’ shap’d it something like a man, 
An’ ca’d it Andrew Turner.
    

В году семьсот сорок девятом 
(Точнее я не помню даты) 
Лепить свинью задумал чёрт. 
Но вдруг в последнее мгновенье 
Он изменил свое решенье, 
И вас он вылепил, милорд!
    

And here is an epigraph of Henry Aldrich:

If on my theme I rightly think,  
There are five reasons why men drink:  
Good wine; a friend; because I'm dry;   
Or lest I should be by and by;  
Or – any other reason why. 
    

Для пьянства есть такие поводы:
Поминки, праздник, встреча, проводы,
Крестины, свадьба и развод,
Мороз, охота, Новый год,
Выздоровленье, новоселье,
Печаль, раскаянье, веселье,
Успех, награда, новый чин
И просто пьянство - без причин! 
    

Links:

• https://soloalgunaspalabras.wordpress.com/2019/01/25/epigrams-and-epitaphs/