TLDR

Problem: trying to connect to a remote server with mosh you see error messages like:

mosh-client needs a UTF-8 native locale to run.
mosh-server needs a UTF-8 native locale to run.
The locale requested by LC_CTYPE=UTF-8 isn't available here.
locale: Cannot set LC_CTYPE to default locale: No such file or directory

OR when connecting with ssh from iTerm2 you get the error message

-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

Solution: If you are not in US/GB/CA/IE/AU/NZ, add the following line to your .profile and restart the shell.

export LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8

I am quite a user of mosh – a mobile shell that makes it really convenient to work via ssh over unstable Internet connection or when roaming across different wifi networks. It’s one of the first tools I install on a new server and from then on I switch from ssh to mosh.

This weekend I was doing an initial configuration to a new server I provisioned from OVH hosting. To my surprise, mosh failed to connect with the following error message:

$ mosh myserver.at.ovh

mosh-client needs a UTF-8 native locale to run.

Unfortunately, the client's environment ([no charset variables]) specifies
the character set "US-ASCII".

LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=
$ 

This was quite a surprise as surely the terminal in MacOS has a UTF-8 native locale. I changed to iTerm2 and tried connecting again, and again ended up with the similar looking error:

$ mosh myserver.at.ovh
/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
The locale requested by LC_CTYPE=UTF-8 isn't available here.
Running `locale-gen UTF-8' may be necessary.

The locale requested by LC_CTYPE=UTF-8 isn't available here.
Running `locale-gen UTF-8' may be necessary.

mosh-server needs a UTF-8 native locale to run.

Unfortunately, the local environment (LC_CTYPE=UTF-8) specifies
the character set "US-ASCII",

The client-supplied environment (LC_CTYPE=UTF-8) specifies
the character set "US-ASCII".

locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=
LC_CTYPE=UTF-8
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
Connection to myserver.at.ovh closed.
/usr/local/bin/mosh: Did not find mosh server startup message. (Have you installed mosh on your server?)
$

Apparently, something was wrong my with locale. Here I realized I have recently done a complete clean re-install and never restored whatever locale settings I had in my .bash_profile.

This actually only increased my curiosity: a freshly installed MacOS with still almost default configuration was causing mosh to fail on connection over locale settings. I googled a bit the mosh error message but didn’t find anything illuminating; then I noticed that when I was connecting to the server with ssh, iTerm2 would emit the following error message:

$ ssh myserver.at.ovh
Last login: Mon Oct  5 11:00:30 2020 from 178.136.74.184
-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

Interestingly, this message wasn’t showing up if I was running ssh from Terminal. So I googled this message in connection with iterm2 and found the issue #5478 with the comment showing an excerpt from iterm2 debug log:

getLocale: languageCode=en, countryCode=ZA
Tentative locale is en_ZA.UTF-8
Locale is NOT supported
Set LC_CTYPE=UTF-8

Now this was illuminating! iTerm2 calculates the value for LC_CTYPE variable by getting the language and country codes from the MacOS. When it ends up with a combination which MacOS lacks locale definition for, it falls back to the default LC_CTYPE=UTF-8. Terminal just defaults to LC_CTYPE=C which is non-UTF8!

Let’s see what English locales MacOS does have the support for:

$ cd /usr/share/locale/
$ ls -1d en_* | column -c 80
en_AU                   en_CA.UTF-8             en_NZ.ISO8859-1
en_AU.ISO8859-1         en_GB                   en_NZ.ISO8859-15
en_AU.ISO8859-15        en_GB.ISO8859-1         en_NZ.US-ASCII
en_AU.US-ASCII          en_GB.ISO8859-15        en_NZ.UTF-8
en_AU.UTF-8             en_GB.US-ASCII          en_US
en_CA                   en_GB.UTF-8             en_US.ISO8859-1
en_CA.ISO8859-1         en_IE                   en_US.ISO8859-15
en_CA.ISO8859-15        en_IE.UTF-8             en_US.US-ASCII
en_CA.US-ASCII          en_NZ                   en_US.UTF-8

Aha, not that many countries are supported, just the Five Eyes and Ireland! That explains why that person from South Africa ran into this issue with en_ZA and iTerm2: every combination of English as the OS language with an odd country will result in it. In my situation, the country was Ukraine with the country code UA, and of course there is no en_UA locale.

How does this affect the remote side of mosh connection – the server I am connecting to, though? Well, ssh – and mosh uses it for authentication phase – by default forwards the LANG and LC_* environment variables to the remote server it is connecting to:

$ grep -n -B1 SendEnv /etc/ssh/ssh_config
48-Host *
49:     SendEnv LANG LC_*

This makes the remote side of the connection declare them, and then mosh-server refuses to start because it requires a UTF-8 native locale to run. When I was connecting from Terminal, the locale was set to C which is not UTF-8, as I mentioned above; when I was connecting from iTerm2, the locale was defaulted to UTF-8 and this also irked mosh-server because there was no definition for it in the system.

How to fix it?

I could generate en_UA.UTF-8 locale definition locally, and that would make terminals define correctly LC_* variables, but they will send them over to the servers I am connecting to, and there mosh-server (and probably some other commands) will fail. This is no good.

Generating en_UA.UTF-8 locale definitions on all the servers I am connecting to is out of question too. Just like declaring overriding LANG/LC_* variables in .bash_profile files on all the servers: this might break legitimate clients with non en_UA.UTF-8 locales.

First working solution would be to disable LANG/LC_* variable forwarding in ssh configuration. Unfortunately, it is set in system-wide /etc/ssh/ssh_config and cannot be overridden in per-user ~/.ssh/config. I hate changing system-wide default setting with no good reason, and the problem with locale didn’t seem to me as a good reason to warrant such change.

Therefore I have opted for another option: just override LANG/LC_* variables in my user’s .bash_profile. This is a more klugy option, but in my situation it looks as the most reasonable thing to do. I have just added the following line to my ~/.bash_profile and it resolved the issue completely:

export LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8