Auto merge of #2691 - micbou:require-utf8-encoding, r=bstaletic

[READY] Require UTF-8 encoding

YCM expects the character encoding used by Vim to be UTF-8. If another encoding is used, inserting a non-ASCII character raises a `UnicodeDecodeError` exception resulting in a Python traceback inside Vim. However, supporting other encodings would make the code more complex for no good reason. A file in a different encoding than UTF-8 can be edited in Vim without any issue when the `encoding` option is set to `utf-8`. For instance, editing a file encoded in `latin-1` is as simple as:
```viml
:e ++enc=latin1
```
There is no need to change the `encoding` option for that.

We don't force `encoding` to be `utf-8` because changing this option is not safe after Vim start. From [the `encoding` documentation](http://vimdoc.sourceforge.net/htmldoc/options.html#'encoding'):
```
NOTE: Changing this option will not change the encoding of the
existing text in Vim.  It may cause non-ASCII text to become invalid.
It should normally be kept at its default value, or set when Vim
starts up.
```
Instead, we require this option to be set to `utf-8` when loading the plugin. If not, we tell users to put the line:
```viml
set encoding=utf-8
```
in their vimrc.

This change should only affect Vim users on Windows where `encoding` is set to `latin1` by default (`encoding` in Neovim is always `utf-8` and cannot be changed). We update the Windows installation guide accordingly.

Closes https://github.com/Valloric/YouCompleteMe/issues/2416.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2691)
<!-- Reviewable:end -->

README.md

@@ -361,6 +361,14 @@ Vim. Look at the features included: `+python/dyn` for Python 2 and
 using a 64-bit client. [Daily updated copies of 32-bit and 64-bit Vim with
 Python 2 and Python 3 support][vim-win-download] are available.
 
+Add the line:
+
+    set encoding=utf-8
+
+to your [vimrc][] if not already present. This option is required by YCM. Note
+that it does not prevent you from editing a file in another encoding than UTF-8.
+You can do that by specifying [the `++enc` argument][++enc] to the `:e` command.
+
 Install YouCompleteMe with [Vundle][].
 
 **Remember:** YCM is a plugin with a compiled component. If you **update** YCM
@@ -3095,3 +3103,4 @@ This software is licensed under the [GPL v3 license][gpl].
 [gitter]: https://gitter.im/Valloric/YouCompleteMe
 [ninja-compdb]: https://ninja-build.org/manual.html
 [vim-nerdtree-tabs]: https://github.com/jistr/vim-nerdtree-tabs
+[++enc]: http://vimdoc.sourceforge.net/htmldoc/editing.html#++enc

doc/youcompleteme.txt

@@ -564,6 +564,15 @@ Vim. Look at the features included: '+python/dyn' for Python 2 and
 using a 64-bit client. Daily updated copies of 32-bit and 64-bit Vim with
 Python 2 and Python 3 support [33] are available.
 
+Add the line:
+>
+  set encoding=utf-8
+<
+to your vimrc [34] if not already present. This option is required by YCM. Note
+that it does not prevent you from editing a file in another encoding than
+UTF-8. You can do that by specifying the '|++enc|' argument to the ':e'
+command.
+
 Install YouCompleteMe with Vundle [23].
 
 **Remember:** YCM is a plugin with a compiled component. If you **update** YCM
@@ -572,16 +581,16 @@ will notify you to recompile it. You should then rerun the install process.
 
 Download and install the following software:
 
-- Python 2 or Python 3 [34]. Be sure to pick the version corresponding to
+- Python 2 or Python 3 [35]. Be sure to pick the version corresponding to
   your Vim architecture. It is _Windows x86_ for a 32-bit Vim and _Windows
   x86-64_ for a 64-bit Vim. We recommend installing Python 3.
 
 - CMake [25]. Add CMake executable to the PATH environment variable.
 
-- Visual Studio [35]. Download the community edition. During setup, select
+- Visual Studio [36]. Download the community edition. During setup, select
   _Desktop development with C++_ in _Workloads_.
 
-- 7-zip [36]. Required to build YCM with semantic support for C-family
+- 7-zip [37]. Required to build YCM with semantic support for C-family
   languages.
 
 Compiling YCM **with** semantic support for C-family languages:
@@ -597,7 +606,7 @@ Compiling YCM **without** semantic support for C-family languages:
 The following additional language support options are available:
 
 - C# support: add '--omnisharp-completer' when calling 'install.py'. Be sure
-  that the build utility 'msbuild' is in your PATH [37].
+  that the build utility 'msbuild' is in your PATH [38].
 
 - Go support: install Go [27] and add '--gocode-completer' when calling
   'install.py'.
@@ -742,9 +751,9 @@ will notify you to recompile it. You should then rerun the install process.
    critical because it must match the Python and the YCM libraries
    architectures. We recommend using a 64-bit Vim.
 
-2. **Install YCM** with Vundle [23] (or Pathogen [38], but Vundle is a
+2. **Install YCM** with Vundle [23] (or Pathogen [39], but Vundle is a
    better idea). With Vundle, this would mean adding a "Plugin
-   'Valloric/YouCompleteMe'" line to your vimrc [39].
+   'Valloric/YouCompleteMe'" line to your vimrc [34].
 
    If you don't install YCM with Vundle, make sure you have run 'git
    submodule update --init --recursive' after checking out the YCM
@@ -781,10 +790,10 @@ will notify you to recompile it. You should then rerun the install process.
    a Debian-like Linux distro, this would be 'sudo apt-get install python-
    dev python3-dev'. On Mac they should already be present.
 
-   On Windows, you need to download and install Python 2 or Python 3 [34].
+   On Windows, you need to download and install Python 2 or Python 3 [35].
    Pick the version corresponding to your Vim architecture. You will also
    need Microsoft Visual C++ (MSVC) to build YCM. You can obtain it by
-   installing Visual Studio [35]. MSVC 12 (Visual Studio 2013), 14 (2015),
+   installing Visual Studio [36]. MSVC 12 (Visual Studio 2013), 14 (2015),
    and 15 (2017) are officially supported.
 
    Here we'll assume you installed YCM with Vundle. That means that the top-
@@ -827,7 +836,7 @@ will notify you to recompile it. You should then rerun the install process.
    extracted the archive file to folder '~/ycm_temp/llvm_root_dir' (with
    'bin', 'lib', 'include' etc. folders right inside that folder). On
    Windows, you can extract the files from the LLVM+Clang installer using
-   7-zip [36].
+   7-zip [37].
 
    **NOTE:** This _only_ works with a _downloaded_ LLVM binary package, not
    a custom-built LLVM! See docs below for 'EXTERNAL_LIBCLANG_PATH' when
@@ -875,7 +884,7 @@ will notify you to recompile it. You should then rerun the install process.
      /property:TargetFrameworkVersion=v4.5
 
    Replace 'msbuild' by 'xbuild' if 'msbuild' is not available. On Windows,
-   be sure that the build utility 'msbuild' is in your PATH [37].
+   be sure that the build utility 'msbuild' is in your PATH [38].
 
    - Go support: install Go [27] and add it to your path. Navigate to
      'YouCompleteMe/third_party/ycmd/third_party/gocode' and run 'go
@@ -1982,11 +1991,11 @@ Options ~
 
 All options have reasonable defaults so if the plug-in works after installation
 you don't need to change any options. These options can be configured in your
-vimrc script [39] by including a line like this:
+vimrc script [34] by including a line like this:
 >
   let g:ycm_min_num_of_chars_for_completion = 1
 <
-Note that after changing an option in your vimrc script [39] you have to
+Note that after changing an option in your vimrc script [34] you have to
 restart Vim for the changes to take effect.
 
 -------------------------------------------------------------------------------
@@ -3350,12 +3359,12 @@ References ~
 [31] http://www.mono-project.com/docs/getting-started/install/linux/#debian-ubuntu-and-derivatives
 [32] http://www.mono-project.com/docs/getting-started/install/linux/#centos-7-fedora-19-and-later-and-derivatives
 [33] https://bintray.com/micbou/generic/vim
-[34] https://www.python.org/downloads/windows/
+[34] http://vimhelp.appspot.com/starting.txt.html#vimrc
-[35] https://www.visualstudio.com/downloads/
+[35] https://www.python.org/downloads/windows/
-[36] http://www.7-zip.org/download.html
+[36] https://www.visualstudio.com/downloads/
-[37] http://stackoverflow.com/questions/6319274/how-do-i-run-msbuild-from-the-command-line-using-windows-sdk-7-1
+[37] http://www.7-zip.org/download.html
-[38] https://github.com/tpope/vim-pathogen#pathogenvim
+[38] http://stackoverflow.com/questions/6319274/how-do-i-run-msbuild-from-the-command-line-using-windows-sdk-7-1
-[39] http://vimhelp.appspot.com/starting.txt.html#vimrc
+[39] https://github.com/tpope/vim-pathogen#pathogenvim
 [40] http://llvm.org/releases/download.html
 [41] http://www.mono-project.com/docs/getting-started/install/
 [42] https://github.com/Valloric/YouCompleteMe#options

plugin/youcompleteme.vim

@@ -29,21 +29,28 @@ if exists( "g:loaded_youcompleteme" )
   finish
 elseif v:version < 704 || (v:version == 704 && !has( 'patch1578' ))
   echohl WarningMsg |
-        \ echomsg "YouCompleteMe unavailable: requires Vim 7.4.1578+" |
+        \ echomsg "YouCompleteMe unavailable: requires Vim 7.4.1578+." |
         \ echohl None
   call s:restore_cpo()
   finish
 elseif !has( 'timers' )
   echohl WarningMsg |
         \ echomsg "YouCompleteMe unavailable: requires Vim compiled with " .
-        \ "the timers feature" |
+        \ "the timers feature." |
         \ echohl None
   call s:restore_cpo()
   finish
 elseif !has( 'python' ) && !has( 'python3' )
   echohl WarningMsg |
         \ echomsg "YouCompleteMe unavailable: requires Vim compiled with " .
-        \ "Python (2.6+ or 3.3+) support" |
+        \ "Python (2.6+ or 3.3+) support." |
+        \ echohl None
+  call s:restore_cpo()
+  finish
+elseif &encoding !~? 'utf-\?8'
+  echohl WarningMsg |
+        \ echomsg "YouCompleteMe unavailable: requires UTF-8 encoding. " .
+        \ "Put the line 'set encoding=utf-8' in your vimrc." |
         \ echohl None
   call s:restore_cpo()
   finish