Auto merge of #2863 - puremourning:java-async-diagnostics, r=micbou

[READY] Java support with asynchronous diagnostics and messages

# PR Prelude

Thank you for working on YCM! :)

**Please complete these steps and check these boxes (by putting an `x` inside
the brackets) _before_ filing your PR:**

- [x] I have read and understood YCM's [CONTRIBUTING][cont] document.
- [x] I have read and understood YCM's [CODE_OF_CONDUCT][code] document.
- [x] I have included tests for the changes in my PR. If not, I have included a
  rationale for why I haven't.
- [x] **I understand my PR may be closed if it becomes obvious I didn't
  actually perform all of these steps.**

# Why this change is necessary and useful

This change is required for a better user experience when using native
java support

This implements an asynchronous message system using a long-poll request
to the server.

The server provides an endpoint /receive_messages which blocks until
either a timeout occurs or we receive a batch of asynchronous messages.
We send this request asynchronously and poll it 4 times a second to see
if we have received any messages.

The messages may either be simply for display (such as startup progress)
or diagnostics, which override the diagnostics returned by
OnFileReqdyToParse.

In the former case, we simply display the message, accepting that this
might be overwritten by any other message (indeed, requiring this), and
for the latter we fan out diagnostics to any open buffer for the file in
question.

Unfortunately, Vim has bugs related to timers when there is something
displayed (such as a "confirm" prompt or other), so we suspend
background timers when doing subcommands to avoid vim bugs. NOTE: This
requires a new version of Vim (detected by the presence of the
particular functions used).

NOT_READY because:

- the submodule commit points at my repo and requires https://github.com/Valloric/ycmd/pull/857 to be merged
- my spider sense suggest i have more testing to do...

Notes:

- Part 3 (I think) of the Java support PRs. This one actually adds the minimal changes for working java support
- There are about 2 or 3 other PRs to come to add things like automatic module imports, etc.

[Please explain **in detail** why the changes in this PR are needed.]

[cont]: https://github.com/Valloric/YouCompleteMe/blob/master/CONTRIBUTING.md
[code]: https://github.com/Valloric/YouCompleteMe/blob/master/CODE_OF_CONDUCT.md

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2863)
<!-- Reviewable:end -->

README.md

@@ -49,6 +49,7 @@ Contents
     - [JavaScript Semantic Completion](#javascript-semantic-completion)
     - [Rust Semantic Completion](#rust-semantic-completion)
     - [Python Semantic Completion](#python-semantic-completion)
+    - [Java Semantic Completion](#java-semantic-completion)
     - [Semantic Completion for Other Languages](#semantic-completion-for-other-languages)
     - [Writing New Semantic Completers](#writing-new-semantic-completers)
     - [Diagnostic Display](#diagnostic-display)
@@ -84,6 +85,7 @@ YouCompleteMe is a fast, as-you-type, fuzzy-search code completion engine for
 - a [TSServer][]-based completion engine for TypeScript,
 - a [Tern][]-based completion engine for JavaScript,
 - a [racer][]-based completion engine for Rust,
+- a [jdt.ls][]-based experimental completion engine for Java.
 - and an omnifunc-based completer that uses data from Vim's omnicomplete system
   to provide semantic completions for many other languages (Ruby, PHP etc.).
 
@@ -217,6 +219,8 @@ The following additional language support options are available:
   `--js-completer` when calling `./install.py`.
 - Rust support: install [Rust][rust-install] and add
   `--rust-completer` when calling `./install.py`.
+- Java support: install [JDK8 (version 8 required)][jdk-install] and add
+  `--java-completer` when calling `./install.py`.
 
 To simply compile with everything enabled, there's a `--all` flag.  So, to
 install with all language features, ensure `xbuild`, `go`, `tsserver`, `node`,
@@ -289,6 +293,8 @@ The following additional language support options are available:
   `--js-completer` when calling `./install.py`.
 - Rust support: install [Rust][rust-install] and add `--rust-completer` when
   calling `./install.py`.
+- Java support: install [JDK8 (version 8 required)][jdk-install] and add
+  `--java-completer` when calling `./install.py`.
 
 To simply compile with everything enabled, there's a `--all` flag.  So, to
 install with all language features, ensure `xbuild`, `go`, `tsserver`, `node`,
@@ -356,6 +362,8 @@ The following additional language support options are available:
   `--js-completer` when calling `./install.py`.
 - Rust support: install [Rust][rust-install] and add `--rust-completer` when
   calling `./install.py`.
+- Java support: install [JDK8 (version 8 required)][jdk-install] and add
+  `--java-completer` when calling `./install.py`.
 
 To simply compile with everything enabled, there's a `--all` flag.  So, to
 install with all language features, ensure `xbuild`, `go`, `tsserver`, `node`,
@@ -449,6 +457,8 @@ The following additional language support options are available:
   `--js-completer` when calling `install.py`.
 - Rust support: install [Rust][rust-install] and add `--rust-completer` when
   calling `install.py`.
+- Java support: install [JDK8 (version 8 required)][jdk-install] and add
+  `--java-completer` when calling `./install.py`.
 
 To simply compile with everything enabled, there's a `--all` flag.  So, to
 install with all language features, ensure `msbuild`, `go`, `tsserver`, `node`,
@@ -520,6 +530,8 @@ The following additional language support options are available:
   `--js-completer` when calling `./install.py`.
 - Rust support: install [Rust][rust-install] and add `--rust-completer` when
   calling `./install.py`.
+- Java support: install [JDK8 (version 8 required)][jdk-install] and add
+  `--java-completer` when calling `./install.py`.
 
 To simply compile with everything enabled, there's a `--all` flag.  So, to
 install with all language features, ensure `xbuild`, `go`, `tsserver`, `node`,
@@ -724,6 +736,13 @@ process.
       `YouCompleteMe/third_party/ycmd/third_party/racerd` and run `cargo build
       --release`.
 
+    - Java support: install [JDK8 (version 8 required)][jdk-install]. Download a
+      [binary release of eclipse.jdt.ls][jdtls-release] and extract it to
+      `YouCompleteMe/third_party/ycmd/third_party/eclipse.jdt.ls/target/repository`.
+      Note: this approach is not recommended for most users and is supported
+      only for advanced users and developers of YCM on a best-efforts basis.
+      Please use `install.py` to enable java support.
+
 That's it. You're done. Refer to the _User Guide_ section on how to use YCM.
 Don't forget that if you want the C-family semantic completion engine to work,
 you will need to provide the compilation flags for your project to YCM. It's all
@@ -802,6 +821,23 @@ Quick Feature Summary
 * Management of `racer` server instance
 * View documentation comments for identifiers (`GetDoc`)
 
+### Java
+
+**NOTE**: Java support is currently experimental. Please let us know your
+[feedback](#contact).
+
+* Semantic auto-completion
+* Go to definition (`GoTo`, `GoToDefinition`, and `GoToDeclaration` are
+  identical)
+* Reference finding (`GoToReferences`)
+* Real-time diagnostic display
+* Renaming symbols (`RefactorRename <new name>`)
+* View documentation comments for identifiers (`GetDoc`)
+* Type information for identifiers (`GetType`)
+* Automatically fix certain errors (`FixIt`)
+* Detection of java projects
+* Management of `jdt.ls` server instance
+
 User Guide
 ----------
 
@@ -995,7 +1031,7 @@ your file.
 
 ### JavaScript Semantic Completion
 
-#### Quick start
+#### JavaScript quick start
 
 1. Ensure that you have enabled the JavaScript completer. See the
    [installation guide](#installation) for details.
@@ -1132,29 +1168,162 @@ in that directory, the first `python` that YCM will find will be the one in the
 virtual environment, so [jedi][] will be able to provide completions for every
 package you have in the virtual environment.
 
+### Java Semantic Completion
+
+**NOTE**: Java support is currently experimental. Please let us know your
+[feedback](#contact).
+
+#### Java quick Start
+
+1. Ensure that you have enabled the Java completer. See the
+   [installation guide](#installation) for details.
+
+2. Create a project file (gradle or maven) file in the root directory of your
+   Java project, by following the instructions below.
+
+3. If you previously used Eclim or Syntastic for Java, disable them for Java.
+
+4. Edit a Java file from your project.
+
+#### Java Project Files
+
+In order to provide semantic analysis, the Java completion engine requires
+knowledge of your project structure. In particular it needs to know the class
+path to use, when compiling your code. Fortunately [jdt.ls][]
+supports [eclipse project files][eclipse-project],
+[maven projects][mvn-project] and [gradle projects][gradle-project].
+
+**NOTE:** Our recommendation is to use either maven or gradle projects.
+
+#### Diagnostic display - Syntastic
+
+The native support for Java includes YCM's native realtime diagnostics display.
+This can conflict with other dianostics plugins like Syntastic, so when enabling
+Java support, please **manually disable Syntastic Java diagnostics**.
+
+Add the following to your `vimrc`:
+
+```viml
+let g:syntastic_java_checkers = []
+```
+
+#### Diagnostic display - Eclim
+
+The native support for Java includes YCM's native realtime diagnostics display.
+This can conflict with other dianostics plugins like Eclim, so when enabling
+Java support, please **manually disable Eclim Java diagnostics**.
+
+Add the following to your `vimrc`:
+
+```viml
+let g:EclimFileTypeValidate = 0
+```
+
+**NOTE**: We recommend disabling Eclim entirely when editing Java with YCM's
+native Java support. This can be done temporarily with `:EclimDisable`.
+
+#### Eclipse Projects
+
+Eclipse style projects require two files: [.project][eclipse-dot-project] and
+[.classpath][eclipse-dot-classpath].
+
+If your project already has these files due to previously being set up within
+eclipse, then no setup is required. [jdt.ls][] should load the project just
+fine (it's basically eclipse after all).
+
+However, if not, it is possible (easy in fact) to craft them manually, though it
+is not recommended. You're better off using gradle or maven (see below).
+
+[A simple eclipse style project example][ycmd-eclipse-project] can be found in
+the ycmd test dir. Normally all that is required is to copy these files to the
+root of your project and to edit the `.classpath` to add additional libraries,
+such as:
+
+```xml
+  <classpathentry kind="lib" path="/path/to/external/jar" />
+  <classpathentry kind="lib" path="/path/to/external/java/source" />
+```
+
+It may also be necessary to change the directory in which your source files are
+located (paths are relative to the .project file itself):
+
+```xml
+  <classpathentry kind="src" output="target/classes" path="path/to/src/" />
+```
+
+**NOTE**: The eclipse project and classpath files are not a public interface
+and it is highly recommended to use Maven or Gradle project definitions if you
+don't already use eclipse to manage your projects.
+
+#### Maven Projects
+
+Maven needs a file named [pom.xml][mvn-project] in the root of the project.
+Once again a simple [pom.xml][ycmd-mvn-pom-xml] can be found in ycmd source.
+
+The format of [pom.xml][mvn-project] files is way beyond the scope of this
+document, but we do recommend using the various tools that can generate them for
+you, if you're not familiar with them already.
+
+#### Gradle Projecs
+
+Gradle projects require a [build.gradle][gradle-project]. Again, there is a
+[trivial example in ycmd's tests][ycmd-gradle-project].
+
+The format of [build.gradle][gradle-project] files is way beyond the scope of
+this document, but we do recommend using the various tools that can generate
+them for you, if you're not familiar with them already.
+
+#### Troubleshooting
+
+If you're not getting completions or diagnostics, check the server health:
+
+* The Java completion engine takes a while to start up and parse your project.
+  You should be able to see its progress in the command line, and
+  `:YcmDebugInfo`. Ensure that the following lines are present:
+
+```
+--   jdt.ls Java Language Server running
+--   jdt.ls Java Language Server Startup Status: Ready
+```
+
+* If the above lines don't appear after a few minutes, check the jdt.ls and ycmd
+  log files using [`:YcmToggleLogs` ](#the-ycmtogglelogs-command). The jdt.ls
+  log file is called `.log` (for some reason).
+
+If you get a message about "classpath is incomplete", then make sure you have
+correctly configured the [project files](#java-project-files).
+
+If you get messages about unresolved imports, then make sure you have
+correctly configured the [project files](#java-project-files), in particular
+check that the classpath is set correctly.
+
+For anything else, [contact us](#contact). Java support is experimental at
+present so we'd love to hear your feedback! Please do remember to check
+[CONTRIBUTING.md][contributing-md] for the list of diagnostics we'll need.
+
 ### Semantic Completion for Other Languages
 
-C-family, C#, Go, JavaScript, Python, Rust, and TypeScript languages are
+C-family, C#, Go, Java, JavaScript, Python, Rust, and TypeScript languages are
 supported natively by YouCompleteMe using the [Clang][], [OmniSharp][],
-[Gocode][]/[Godef][], [Tern][], [Jedi][], [racer][], and [TSServer][] engines,
+[Gocode][]/[Godef][], [jdt.ls][], [Tern][], [Jedi][], [racer][], and
-respectively. Check the [installation](#installation) section for instructions
+[TSServer][] engines, respectively. Check the [installation](#installation)
-to enable these features if desired.
+section for instructions to enable these features if desired.
 
 YCM will use your `omnifunc` (see `:h omnifunc` in Vim) as a source for semantic
 completions if it does not have a native semantic completion engine for your
 file's filetype. Vim comes with okayish omnifuncs for various languages like
 Ruby, PHP, etc. It depends on the language.
 
-You can get stellar omnifuncs for Java and Ruby with [Eclim][]. Just make sure
+You can get a stellar omnifunc for Ruby with [Eclim][]. Just make sure you have
-you have the _latest_ Eclim installed and configured (this means Eclim `>= 2.2.*`
+the _latest_ Eclim installed and configured (this means Eclim `>= 2.2.*` and
-and Eclipse `>= 4.2.*`).
+Eclipse `>= 4.2.*`).
 
 After installing Eclim remember to create a new Eclipse project within your
-application by typing `:ProjectCreate <path-to-your-project> -n ruby` (or `-n java`)
+application by typing `:ProjectCreate <path-to-your-project> -n ruby` inside vim
-inside vim and don't forget to have `let g:EclimCompletionMethod = 'omnifunc'`
+and don't forget to have `let g:EclimCompletionMethod = 'omnifunc'` in your
-in your vimrc. This will make YCM and Eclim play nice; YCM will use Eclim's omnifuncs
+vimrc. This will make YCM and Eclim play nice; YCM will use Eclim's omnifuncs as
-as the data source for semantic completions and provide the auto-triggering
+the data source for semantic completions and provide the auto-triggering and
-and subsequence-based matching (and other YCM features) on top of it.
+subsequence-based matching (and other YCM features) on top of it.
 
 ### Writing New Semantic Completers
 
@@ -1372,7 +1541,7 @@ Supported in filetypes: `c, cpp, objc, objcpp`
 
 Looks up the symbol under the cursor and jumps to its declaration.
 
-Supported in filetypes: `c, cpp, objc, objcpp, cs, go, python, rust`
+Supported in filetypes: `c, cpp, objc, objcpp, cs, go, python, rust, java`
 
 #### The `GoToDefinition` subcommand
 
@@ -1384,7 +1553,7 @@ translation unit consists of the file you are editing and all the files you are
 including with `#include` directives (directly or indirectly) in that file.
 
 Supported in filetypes: `c, cpp, objc, objcpp, cs, go, javascript, python,
-rust, typescript`
+rust, typescript, java`
 
 #### The `GoTo` subcommand
 
@@ -1395,7 +1564,8 @@ the current translation unit, jumps to the symbol's declaration. For
 C/C++/Objective-C, it first tries to look up the current line for a header and
 jump to it. For C#, implementations are also considered and preferred.
 
-Supported in filetypes: `c, cpp, objc, objcpp, cs, go, javascript, python, rust`
+Supported in filetypes: `c, cpp, objc, objcpp, cs, go, javascript, python, rust,
+java`
 
 #### The `GoToImprecise` subcommand
 
@@ -1416,7 +1586,7 @@ This command attempts to find all of the references within the project to the
 identifier under the cursor and populates the quickfix list with those
 locations.
 
-Supported in filetypes: `javascript, python, typescript`
+Supported in filetypes: `javascript, python, typescript, java`
 
 #### The `GoToImplementation` subcommand
 
@@ -1424,7 +1594,7 @@ Looks up the symbol under the cursor and jumps to its implementation (i.e.
 non-interface). If there are multiple implementations, instead provides a list
 of implementations to choose from.
 
-Supported in filetypes: `cs`
+Supported in filetypes: `cs, java`
 
 #### The `GoToImplementationElseDeclaration` subcommand
 
@@ -1454,7 +1624,7 @@ Invoking this command on `s` returns `std::string => std::basic_string<char>`
 
 **NOTE:** Causes re-parsing of the current translation unit.
 
-Supported in filetypes: `c, cpp, objc, objcpp, javascript, typescript`
+Supported in filetypes: `c, cpp, objc, objcpp, javascript, typescript, java`
 
 #### The `GetTypeImprecise` subcommand
 
@@ -1511,7 +1681,7 @@ under the cursor. Depending on the file type, this includes things like:
 * etc.
 
 Supported in filetypes: `c, cpp, objc, objcpp, cs, python, typescript,
-javascript, rust`
+javascript, rust, java`
 
 #### The `GetDocImprecise` subcommand
 
@@ -1559,7 +1729,7 @@ indication).
 
 **NOTE:** Causes re-parsing of the current translation unit.
 
-Supported in filetypes: `c, cpp, objc, objcpp, cs`
+Supported in filetypes: `c, cpp, objc, objcpp, cs, java`
 
 #### The `RefactorRename <new name>` subcommand
 
@@ -1573,7 +1743,7 @@ files. Rename operations may involve changes to multiple files, which may or may
 not be open in Vim buffers at the time. YouCompleteMe handles all of this for
 you. The behavior is described in [the following section](#multi-file-refactor).
 
-Supported in filetypes: `javascript` (variables only), `typescript`
+Supported in filetypes: `javascript` (variables only), `typescript, java`
 
 #### Multi-file Refactor
 
@@ -1626,7 +1796,7 @@ python binary to use to restart the Python semantic engine.
 :YcmCompleter RestartServer /usr/bin/python3.4
 ```
 
-Supported in filetypes: `cs, go, javascript, python, rust, typescript`
+Supported in filetypes: `cs, go, javascript, python, rust, typescript, java`
 
 #### The `ClearCompilationFlagCache` subcommand
 
@@ -3189,3 +3359,14 @@ This software is licensed under the [GPL v3 license][gpl].
 [++enc]: http://vimdoc.sourceforge.net/htmldoc/editing.html#++enc
 [rustup]: https://www.rustup.rs/
 [contributing-md]: https://github.com/Valloric/YouCompleteMe/blob/master/CONTRIBUTING.md
+[jdt.ls]: https://github.com/eclipse/eclipse.jdt.ls
+[jdk-install]: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
+[mvn-project]: https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html
+[eclipse-project]: https://help.eclipse.org/oxygen/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Freference%2Fmisc%2Fproject_description_file.html
+[gradle-project]: https://docs.gradle.org/current/userguide/tutorial_java_projects.html
+[eclipse-dot-project]: https://help.eclipse.org/oxygen/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Freference%2Fmisc%2Fproject_description_file.html
+[eclipse-dot-classpath]: https://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2FIClasspathEntry.html
+[ycmd-eclipse-project]: https://github.com/Valloric/ycmd/tree/master/ycmd/tests/java/testdata/simple_eclipse_project
+[ycmd-mvn-pom-xml]: https://github.com/Valloric/ycmd/blob/java-language-server/ycmd/tests/java/testdata/simple_maven_project/pom.xml
+[ycmd-gradle-project]: https://github.com/Valloric/ycmd/tree/master/ycmd/tests/java/testdata/simple_gradle_project
+[jdtls-release]: http://download.eclipse.org/jdtls/milestones

autoload/youcompleteme.vim

@@ -41,6 +41,10 @@ let s:pollers = {
       \   'server_ready': {
       \     'id': -1,
       \     'wait_milliseconds': 100
+      \   },
+      \   'receive_messages': {
+      \     'id': -1,
+      \     'wait_milliseconds': 100
       \   }
       \ }
 
@@ -71,6 +75,29 @@ function! s:Pyeval( eval_string )
 endfunction
 
 
+function! s:StartMessagePoll()
+  if s:pollers.receive_messages.id < 0
+    let s:pollers.receive_messages.id = timer_start(
+          \ s:pollers.receive_messages.wait_milliseconds,
+          \ function( 's:ReceiveMessages' ) )
+  endif
+endfunction
+
+
+function! s:ReceiveMessages( timer_id )
+  let poll_again = s:Pyeval( 'ycm_state.OnPeriodicTick()' )
+
+  if poll_again
+    let s:pollers.receive_messages.id = timer_start(
+          \ s:pollers.receive_messages.wait_milliseconds,
+          \ function( 's:ReceiveMessages' ) )
+  else
+    " Don't poll again until we open another buffer
+    let s:pollers.receive_messages.id = -1
+  endif
+endfunction
+
+
 function! youcompleteme#Enable()
   call s:SetUpBackwardsCompatibility()
 
@@ -451,6 +478,7 @@ function! s:OnFileTypeSet()
 
   call s:SetUpCompleteopt()
   call s:SetCompleteFunc()
+  call s:StartMessagePoll()
 
   exec s:python_command "ycm_state.OnBufferVisit()"
   call s:OnFileReadyToParse( 1 )
@@ -464,6 +492,7 @@ function! s:OnBufferEnter()
 
   call s:SetUpCompleteopt()
   call s:SetCompleteFunc()
+  call s:StartMessagePoll()
 
   exec s:python_command "ycm_state.OnBufferVisit()"
   " Last parse may be outdated because of changes from other buffers. Force a
@@ -801,6 +830,10 @@ endfunction
 
 function! s:RestartServer()
   exec s:python_command "ycm_state.RestartServer()"
+
+  call timer_stop( s:pollers.receive_messages.id )
+  let s:pollers.receive_messages.id = -1
+
   call timer_stop( s:pollers.server_ready.id )
   let s:pollers.server_ready.id = timer_start(
         \ s:pollers.server_ready.wait_milliseconds,
@@ -828,11 +861,11 @@ endfunction
 
 
 function! s:CompleterCommand(...)
-  " CompleterCommand will call the OnUserCommand function of a completer.
+  " CompleterCommand will call the OnUserCommand function of a completer.  If
-  " If the first arguments is of the form "ft=..." it can be used to specify the
+  " the first arguments is of the form "ft=..." it can be used to specify the
-  " completer to use (for example "ft=cpp").  Else the native filetype completer
+  " completer to use (for example "ft=cpp").  Else the native filetype
-  " of the current buffer is used.  If no native filetype completer is found and
+  " completer of the current buffer is used.  If no native filetype completer
-  " no completer was specified this throws an error.  You can use
+  " is found and no completer was specified this throws an error.  You can use
   " "ft=ycm:ident" to select the identifier completer.
   " The remaining arguments will be passed to the completer.
   let arguments = copy(a:000)

python/ycm/buffer.py

@@ -27,17 +27,23 @@ from ycm.client.event_notification import EventNotification
 from ycm.diagnostic_interface import DiagnosticInterface
 
 
+DIAGNOSTIC_UI_FILETYPES = set( [ 'cpp', 'cs', 'c', 'objc', 'objcpp',
+                                 'typescript' ] )
+DIAGNOSTIC_UI_ASYNC_FILETYPES = set( [ 'java' ] )
+
+
 # Emulates Vim buffer
 # Used to store buffer related information like diagnostics, latest parse
 # request. Stores buffer change tick at the parse request moment, allowing
 # to effectively determine whether reparse is needed for the buffer.
 class Buffer( object ):
 
-  def __init__( self, bufnr, user_options ):
+  def __init__( self, bufnr, user_options, async_diags ):
     self.number = bufnr
     self._parse_tick = 0
     self._handled_tick = 0
     self._parse_request = None
+    self._async_diags = async_diags
     self._diag_interface = DiagnosticInterface( bufnr, user_options )
 
 
@@ -60,9 +66,18 @@ class Buffer( object ):
     return self._parse_tick != self._ChangedTick()
 
 
-  def UpdateDiagnostics( self ):
+  def UpdateDiagnostics( self, force=False ):
-    self._diag_interface.UpdateWithNewDiagnostics(
+    if force or not self._async_diags:
-      self._parse_request.Response() )
+      self.UpdateWithNewDiagnostics( self._parse_request.Response() )
+    else:
+      # We need to call the response method, because it might throw an exception
+      # or require extra config confirmation, even if we don't actually use the
+      # diagnostics.
+      self._parse_request.Response()
+
+
+  def UpdateWithNewDiagnostics( self, diagnostics ):
+    self._diag_interface.UpdateWithNewDiagnostics( diagnostics )
 
 
   def PopulateLocationList( self ):
@@ -105,5 +120,11 @@ class BufferDict( dict ):
 
   def __missing__( self, key ):
     # Python does not allow to return assignment operation result directly
-    new_value = self[ key ] = Buffer( key, self._user_options )
+    new_value = self[ key ] = Buffer(
+      key,
+      self._user_options,
+      any( [ x in DIAGNOSTIC_UI_ASYNC_FILETYPES
+             for x in
+             vimsupport.GetBufferFiletypes( key ) ] ) )
+
     return new_value

python/ycm/client/messages_request.py

@@ -0,0 +1,97 @@
+# Copyright (C) 2017 YouCompleteMe contributors
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import unicode_literals
+from __future__ import print_function
+from __future__ import division
+from __future__ import absolute_import
+# Not installing aliases from python-future; it's unreliable and slow.
+from builtins import *  # noqa
+
+from ycm.vimsupport import PostVimMessage
+
+from ycm.client.base_request import ( BaseRequest, BuildRequestData,
+                                      JsonFromFuture, HandleServerException )
+
+import logging
+
+_logger = logging.getLogger( __name__ )
+
+# Looooong poll
+TIMEOUT_SECONDS = 60
+
+
+class MessagesPoll( BaseRequest ):
+  def __init__( self ):
+    super( MessagesPoll, self ).__init__()
+    self._request_data = BuildRequestData()
+    self._response_future = None
+
+
+  def _SendRequest( self ):
+    self._response_future = self.PostDataToHandlerAsync(
+      self._request_data,
+      'receive_messages',
+      timeout = TIMEOUT_SECONDS )
+    return
+
+
+  def Poll( self, diagnostics_handler ):
+    """This should be called regularly to check for new messages in this buffer.
+    Returns True if Poll should be called again in a while. Returns False when
+    the completer or server indicated that further polling should not be done
+    for the requested file."""
+
+    if self._response_future is None:
+      # First poll
+      self._SendRequest()
+      return True
+
+    if not self._response_future.done():
+      # Nothing yet...
+      return True
+
+    with HandleServerException( display = False ):
+      response = JsonFromFuture( self._response_future )
+
+      poll_again = _HandlePollResponse( response, diagnostics_handler )
+      if poll_again:
+        self._SendRequest()
+        return True
+
+    return False
+
+
+def _HandlePollResponse( response, diagnostics_handler ):
+  if isinstance( response, list ):
+    for notification in response:
+      if 'message' in notification:
+        PostVimMessage( notification[ 'message' ],
+                        warning = False,
+                        truncate = True )
+      elif 'diagnostics' in notification:
+        diagnostics_handler.UpdateWithNewDiagnosticsForFile(
+          notification[ 'filepath' ],
+          notification[ 'diagnostics' ] )
+  elif response is False:
+    # Don't keep polling for this file
+    return False
+  # else any truthy response means "nothing to see here; poll again in a
+  # while"
+
+  # Start the next poll (only if the last poll didn't raise an exception)
+  return True

python/ycm/diagnostic_interface.py

@@ -123,7 +123,8 @@ class DiagnosticInterface( object ):
 
 
   def _UpdateLocationList( self ):
-    vimsupport.SetLocationList(
+    vimsupport.SetLocationListForBuffer(
+      self._bufnr,
       vimsupport.ConvertDiagnosticsToQfList( self._diagnostics ) )
 
 

python/ycm/tests/client/messages_request_test.py

@@ -0,0 +1,142 @@
+# Copyright (C) 2017 YouCompleteMe Contributors
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+
+from __future__ import unicode_literals
+from __future__ import print_function
+from __future__ import division
+from __future__ import absolute_import
+# Not installing aliases from python-future; it's unreliable and slow.
+from builtins import *  # noqa
+
+from ycm.tests.test_utils import MockVimModule
+MockVimModule()
+
+from hamcrest import assert_that, equal_to
+from mock import patch, call
+
+from ycm.client.messages_request import _HandlePollResponse
+from ycm.tests.test_utils import ExtendedMock
+
+
+def HandlePollResponse_NoMessages_test():
+  assert_that( _HandlePollResponse( True, None ), equal_to( True ) )
+
+  # Other non-False responses mean the same thing
+  assert_that( _HandlePollResponse( '', None ), equal_to( True ) )
+  assert_that( _HandlePollResponse( 1, None ), equal_to( True ) )
+  assert_that( _HandlePollResponse( {}, None ), equal_to( True ) )
+
+
+def HandlePollResponse_PollingNotSupported_test():
+  assert_that( _HandlePollResponse( False, None ), equal_to( False ) )
+
+  # 0 is not False
+  assert_that( _HandlePollResponse( 0, None ), equal_to( True ) )
+
+
+@patch( 'ycm.client.messages_request.PostVimMessage',
+        new_callable = ExtendedMock )
+def HandlePollResponse_SingleMessage_test( post_vim_message ):
+  assert_that( _HandlePollResponse( [ { 'message': 'this is a message' } ] ,
+                                    None ),
+               equal_to( True ) )
+
+  post_vim_message.assert_has_exact_calls( [
+    call( 'this is a message', warning=False, truncate=True )
+  ] )
+
+
+@patch( 'ycm.client.messages_request.PostVimMessage',
+        new_callable = ExtendedMock )
+def HandlePollResponse_MultipleMessages_test( post_vim_message ):
+  assert_that( _HandlePollResponse( [ { 'message': 'this is a message' },
+                                      { 'message': 'this is another one' } ] ,
+                                    None ),
+               equal_to( True ) )
+
+  post_vim_message.assert_has_exact_calls( [
+    call( 'this is a message', warning=False, truncate=True ),
+    call( 'this is another one', warning=False, truncate=True )
+  ] )
+
+
+def HandlePollResponse_SingleDiagnostic_test():
+  diagnostics_handler = ExtendedMock()
+  messages = [
+    { 'filepath': 'foo', 'diagnostics': [ 'PLACEHOLDER' ] },
+  ]
+  assert_that( _HandlePollResponse( messages, diagnostics_handler ),
+               equal_to( True ) )
+  diagnostics_handler.UpdateWithNewDiagnosticsForFile.assert_has_exact_calls( [
+    call( 'foo', [ 'PLACEHOLDER' ] )
+  ] )
+
+
+def HandlePollResponse_MultipleDiagnostics_test():
+  diagnostics_handler = ExtendedMock()
+  messages = [
+    { 'filepath': 'foo', 'diagnostics': [ 'PLACEHOLDER1' ] },
+    { 'filepath': 'bar', 'diagnostics': [ 'PLACEHOLDER2' ] },
+    { 'filepath': 'baz', 'diagnostics': [ 'PLACEHOLDER3' ] },
+    { 'filepath': 'foo', 'diagnostics': [ 'PLACEHOLDER4' ] },
+  ]
+  assert_that( _HandlePollResponse( messages, diagnostics_handler ),
+               equal_to( True ) )
+  diagnostics_handler.UpdateWithNewDiagnosticsForFile.assert_has_exact_calls( [
+    call ( 'foo', [ 'PLACEHOLDER1' ] ),
+    call ( 'bar', [ 'PLACEHOLDER2' ] ),
+    call ( 'baz', [ 'PLACEHOLDER3' ] ),
+    call ( 'foo', [ 'PLACEHOLDER4' ] )
+  ] )
+
+
+@patch( 'ycm.client.messages_request.PostVimMessage',
+        new_callable = ExtendedMock )
+def HandlePollResponse_MultipleMessagesAndDiagnostics_test( post_vim_message ):
+  diagnostics_handler = ExtendedMock()
+  messages = [
+    { 'filepath': 'foo', 'diagnostics': [ 'PLACEHOLDER1' ] },
+    { 'message': 'On the first day of Christmas, my VimScript gave to me' },
+    { 'filepath': 'bar', 'diagnostics': [ 'PLACEHOLDER2' ] },
+    { 'message': 'A test file in a Command-T' },
+    { 'filepath': 'baz', 'diagnostics': [ 'PLACEHOLDER3' ] },
+    { 'message': 'On the second day of Christmas, my VimScript gave to me' },
+    { 'filepath': 'foo', 'diagnostics': [ 'PLACEHOLDER4' ] },
+    { 'message': 'Two popup menus, and a test file in a Command-T' },
+  ]
+  assert_that( _HandlePollResponse( messages, diagnostics_handler ),
+               equal_to( True ) )
+  diagnostics_handler.UpdateWithNewDiagnosticsForFile.assert_has_exact_calls( [
+    call ( 'foo', [ 'PLACEHOLDER1' ] ),
+    call ( 'bar', [ 'PLACEHOLDER2' ] ),
+    call ( 'baz', [ 'PLACEHOLDER3' ] ),
+    call ( 'foo', [ 'PLACEHOLDER4' ] )
+  ] )
+
+  post_vim_message.assert_has_exact_calls( [
+    call( 'On the first day of Christmas, my VimScript gave to me',
+          warning=False,
+          truncate=True ),
+    call( 'A test file in a Command-T', warning=False, truncate=True ),
+    call( 'On the second day of Christmas, my VimScript gave to me',
+          warning=False,
+          truncate=True ),
+    call( 'Two popup menus, and a test file in a Command-T',
+          warning=False,
+          truncate=True ),
+  ] )

python/ycm/tests/mock_utils.py

@@ -0,0 +1,113 @@
+# Copyright (C) 2017 YouCompleteMe contributors
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+from __future__ import unicode_literals
+from __future__ import print_function
+from __future__ import division
+from __future__ import absolute_import
+# Not installing aliases from python-future; it's unreliable and slow.
+from builtins import *  # noqa
+
+import mock
+import requests
+
+
+class FakeResponse( object ):
+  """A fake version of a requests response object, just about suitable for
+  mocking a server response. Not usually used directly. See
+  MockServerResponse* methods"""
+  def __init__( self, response, exception ):
+    self._json = response
+    self._exception = exception
+    self.status_code = requests.codes.ok
+    self.text = not exception
+
+  def json( self ):
+    if self._exception:
+      return None
+    return self._json
+
+
+  def raise_for_status( self ):
+    if self._exception:
+      raise self._exception
+
+
+class FakeFuture( object ):
+  """A fake version of a future response object, just about suitable for
+  mocking a server response as generated by PostDataToHandlerAsync.
+  Not usually used directly. See MockAsyncServerResponse* methods"""
+  def __init__( self, done, response = None, exception = None ):
+    self._done = done
+
+    if not done:
+      self._result = None
+    else:
+      self._result = FakeResponse( response, exception )
+
+
+  def done( self ):
+    return self._done
+
+
+  def result( self ):
+    return self._result
+
+
+def MockAsyncServerResponseDone( response ):
+  """Return a fake future object that is complete with the supplied response
+  message. Suitable for mocking a response future within a client request. For
+  example:
+
+    server_message = {
+      'message': 'this message came from the server'
+    }
+
+    with patch.object( ycm._message_poll_request,
+                       '_response_future',
+                       new = MockAsyncServerResponseDone( [] ) ) as mock_future:
+      ycm.OnPeriodicTick() # Uses ycm._message_poll_request ...
+  """
+  return mock.MagicMock( wraps = FakeFuture( True, response ) )
+
+
+def MockAsyncServerResponseInProgress():
+  """Return a fake future object that is incomplete. Suitable for mocking a
+  response future within a client request. For example:
+
+    with patch.object( ycm._message_poll_request,
+                       '_response_future',
+                       new = MockAsyncServerResponseInProgress() ):
+      ycm.OnPeriodicTick() # Uses ycm._message_poll_request ...
+  """
+  return mock.MagicMock( wraps = FakeFuture( False ) )
+
+
+def MockAsyncServerResponseException( exception ):
+  """Return a fake future object that is complete, but raises an exception.
+  Suitable for mocking a response future within a client request. For example:
+
+    exception = RuntimeError( 'Check client handles exception' )
+    with patch.object( ycm._message_poll_request,
+                       '_response_future',
+                       new = MockAsyncServerResponseException( exception ) ):
+      ycm.OnPeriodicTick() # Uses ycm._message_poll_request ...
+  """
+  return mock.MagicMock( wraps = FakeFuture( True, None, exception ) )
+
+
+# TODO: In future, implement MockServerResponse and MockServerResponseException
+# for synchronous cases when such test cases are needed.

python/ycm/tests/vimsupport_test.py

@@ -39,6 +39,74 @@ import os
 import json
 
 
+@patch( 'vim.eval', new_callable = ExtendedMock )
+def SetLocationListForBuffer_Current_test( vim_eval ):
+  diagnostics = [ {
+    'bufnr': 3,
+    'filename': 'some_filename',
+    'lnum': 5,
+    'col': 22,
+    'type': 'E',
+    'valid': 1
+  } ]
+  current_buffer = VimBuffer( '/test', number = 3, window = 7 )
+  with MockVimBuffers( [ current_buffer ], current_buffer, ( 1, 1 ) ):
+    vimsupport.SetLocationListForBuffer( 3, diagnostics )
+
+  # We asked for the buffer which is current, so we use winnr 0
+  vim_eval.assert_has_exact_calls( [
+    call( 'setloclist( 0, {0} )'.format( json.dumps( diagnostics ) ) )
+  ] )
+
+
+@patch( 'vim.eval', new_callable = ExtendedMock, side_effect = [ 8, 1 ] )
+def SetLocationListForBuffer_NotCurrent_test( vim_eval ):
+  diagnostics = [ {
+    'bufnr': 3,
+    'filename': 'some_filename',
+    'lnum': 5,
+    'col': 22,
+    'type': 'E',
+    'valid': 1
+  } ]
+  current_buffer = VimBuffer( '/test', number = 3, window = 7 )
+  other_buffer = VimBuffer( '/notcurrent', number = 1, window = 8 )
+  with MockVimBuffers( [ current_buffer, other_buffer ],
+                       current_buffer,
+                       ( 1, 1 ) ):
+    vimsupport.SetLocationListForBuffer( 1, diagnostics )
+
+  # We asked for a buffer which is not current, so we find the window
+  vim_eval.assert_has_exact_calls( [
+    call( 'bufwinnr(1)' ), # returns 8 due to side_effect
+    call( 'setloclist( 8, {0} )'.format( json.dumps( diagnostics ) ) )
+  ] )
+
+
+@patch( 'vim.eval', new_callable = ExtendedMock, side_effect = [ -1, 1 ] )
+def SetLocationListForBuffer_NotVisible_test( vim_eval ):
+  diagnostics = [ {
+    'bufnr': 3,
+    'filename': 'some_filename',
+    'lnum': 5,
+    'col': 22,
+    'type': 'E',
+    'valid': 1
+  } ]
+  current_buffer = VimBuffer( '/test', number = 3, window = 7 )
+  other_buffer = VimBuffer( '/notcurrent', number = 1, window = 8 )
+  with MockVimBuffers( [ current_buffer, other_buffer ],
+                       current_buffer,
+                       ( 1, 1 ) ):
+    vimsupport.SetLocationListForBuffer( 1, diagnostics )
+
+  # We asked for a buffer which is not current, so we find the window
+  vim_eval.assert_has_exact_calls( [
+    call( 'bufwinnr(1)' ), # returns -1 due to side_effect
+    call( 'setloclist( 0, {0} )'.format( json.dumps( diagnostics ) ) )
+  ] )
+
+
 @patch( 'vim.eval', new_callable = ExtendedMock )
 def SetLocationList_test( vim_eval ):
   diagnostics = [ {
@@ -49,9 +117,36 @@ def SetLocationList_test( vim_eval ):
     'type': 'E',
     'valid': 1
   } ]
-  vimsupport.SetLocationList( diagnostics )
+  current_buffer = VimBuffer( '/test', number = 3, window = 7 )
-  vim_eval.assert_called_once_with(
+  with MockVimBuffers( [ current_buffer ], current_buffer, ( 1, 1 ) ):
-    'setloclist( 0, {0} )'.format( json.dumps( diagnostics ) ) )
+    vimsupport.SetLocationList( diagnostics )
+
+  vim_eval.assert_has_calls( [
+    call( 'setloclist( 0, {0} )'.format( json.dumps( diagnostics ) ) ),
+  ] )
+
+
+@patch( 'vim.eval', new_callable = ExtendedMock )
+def SetLocationList_NotCurrent_test( vim_eval ):
+  diagnostics = [ {
+    'bufnr': 3,
+    'filename': 'some_filename',
+    'lnum': 5,
+    'col': 22,
+    'type': 'E',
+    'valid': 1
+  } ]
+  current_buffer = VimBuffer( '/test', number = 3, window = 7 )
+  other_buffer = VimBuffer( '/notcurrent', number = 1, window = 8 )
+  with MockVimBuffers( [ current_buffer, other_buffer ],
+                       current_buffer,
+                       ( 1, 1 ) ):
+    vimsupport.SetLocationList( diagnostics )
+
+  # This version does not check the current buffer and just sets the current win
+  vim_eval.assert_has_exact_calls( [
+    call( 'setloclist( 0, {0} )'.format( json.dumps( diagnostics ) ) ),
+  ] )
 
 
 @patch( 'ycm.vimsupport.VariableExists', return_value = True )

python/ycm/tests/youcompleteme_test.py

@@ -38,6 +38,12 @@ from ycm.tests import ( MakeUserOptions, StopServer, test_utils,
                         WaitUntilReady, YouCompleteMeInstance )
 from ycm.client.base_request import _LoadExtraConfFile
 from ycmd.responses import ServerError
+from ycm.tests.mock_utils import ( MockAsyncServerResponseDone,
+                                   MockAsyncServerResponseInProgress,
+                                   MockAsyncServerResponseException )
+
+
+from ycm import buffer as ycm_buffer_module
 
 
 @YouCompleteMeInstance()
@@ -394,11 +400,11 @@ def YouCompleteMe_ShowDiagnostics_FiletypeNotSupported_test( ycm,
 @patch( 'ycm.youcompleteme.YouCompleteMe.FiletypeCompleterExistsForFiletype',
         return_value = True )
 @patch( 'ycm.vimsupport.PostVimMessage', new_callable = ExtendedMock )
-@patch( 'ycm.vimsupport.SetLocationList', new_callable = ExtendedMock )
+@patch( 'ycm.vimsupport.SetLocationListForWindow', new_callable = ExtendedMock )
 def YouCompleteMe_ShowDiagnostics_NoDiagnosticsDetected_test(
-  ycm, set_location_list, post_vim_message, *args ):
+  ycm, set_location_list_for_window, post_vim_message, *args ):
 
-  current_buffer = VimBuffer( 'buffer', filetype = 'cpp' )
+  current_buffer = VimBuffer( 'buffer', filetype = 'cpp', window = 99 )
   with MockVimBuffers( [ current_buffer ], current_buffer ):
     with patch( 'ycm.client.event_notification.EventNotification.Response',
                 return_value = {} ):
@@ -410,7 +416,7 @@ def YouCompleteMe_ShowDiagnostics_NoDiagnosticsDetected_test(
     call( 'Diagnostics refreshed', warning = False ),
     call( 'No warnings or errors detected.', warning = False )
   ] )
-  set_location_list.assert_called_once_with( [] )
+  set_location_list_for_window.assert_called_once_with( 0, [] )
 
 
 @YouCompleteMeInstance( { 'log_level': 'debug',
@@ -419,9 +425,9 @@ def YouCompleteMe_ShowDiagnostics_NoDiagnosticsDetected_test(
 @patch( 'ycm.youcompleteme.YouCompleteMe.FiletypeCompleterExistsForFiletype',
         return_value = True )
 @patch( 'ycm.vimsupport.PostVimMessage', new_callable = ExtendedMock )
-@patch( 'ycm.vimsupport.SetLocationList', new_callable = ExtendedMock )
+@patch( 'ycm.vimsupport.SetLocationListForWindow', new_callable = ExtendedMock )
 def YouCompleteMe_ShowDiagnostics_DiagnosticsFound_DoNotOpenLocationList_test(
-  ycm, set_location_list, post_vim_message, *args ):
+  ycm, set_location_list_for_window, post_vim_message, *args ):
 
   diagnostic = {
     'kind': 'ERROR',
@@ -433,7 +439,10 @@ def YouCompleteMe_ShowDiagnostics_DiagnosticsFound_DoNotOpenLocationList_test(
     }
   }
 
-  current_buffer = VimBuffer( 'buffer', filetype = 'cpp', number = 3 )
+  current_buffer = VimBuffer( 'buffer',
+                              filetype = 'cpp',
+                              number = 3,
+                              window = 99 )
   with MockVimBuffers( [ current_buffer ], current_buffer ):
     with patch( 'ycm.client.event_notification.EventNotification.Response',
                 return_value = [ diagnostic ] ):
@@ -444,7 +453,7 @@ def YouCompleteMe_ShowDiagnostics_DiagnosticsFound_DoNotOpenLocationList_test(
           warning = False ),
     call( 'Diagnostics refreshed', warning = False )
   ] )
-  set_location_list.assert_called_once_with( [ {
+  set_location_list_for_window.assert_called_once_with( 0, [ {
       'bufnr': 3,
       'lnum': 19,
       'col': 2,
@@ -458,10 +467,14 @@ def YouCompleteMe_ShowDiagnostics_DiagnosticsFound_DoNotOpenLocationList_test(
 @patch( 'ycm.youcompleteme.YouCompleteMe.FiletypeCompleterExistsForFiletype',
         return_value = True )
 @patch( 'ycm.vimsupport.PostVimMessage', new_callable = ExtendedMock )
-@patch( 'ycm.vimsupport.SetLocationList', new_callable = ExtendedMock )
+@patch( 'ycm.vimsupport.SetLocationListForWindow', new_callable = ExtendedMock )
 @patch( 'ycm.vimsupport.OpenLocationList', new_callable = ExtendedMock )
 def YouCompleteMe_ShowDiagnostics_DiagnosticsFound_OpenLocationList_test(
-  ycm, open_location_list, set_location_list, post_vim_message, *args ):
+  ycm,
+  open_location_list,
+  set_location_list_for_window,
+  post_vim_message,
+  *args ):
 
   diagnostic = {
     'kind': 'ERROR',
@@ -473,7 +486,10 @@ def YouCompleteMe_ShowDiagnostics_DiagnosticsFound_OpenLocationList_test(
     }
   }
 
-  current_buffer = VimBuffer( 'buffer', filetype = 'cpp', number = 3 )
+  current_buffer = VimBuffer( 'buffer',
+                              filetype = 'cpp',
+                              number = 3,
+                              window = 99 )
   with MockVimBuffers( [ current_buffer ], current_buffer ):
     with patch( 'ycm.client.event_notification.EventNotification.Response',
                 return_value = [ diagnostic ] ):
@@ -484,7 +500,7 @@ def YouCompleteMe_ShowDiagnostics_DiagnosticsFound_OpenLocationList_test(
           warning = False ),
     call( 'Diagnostics refreshed', warning = False )
   ] )
-  set_location_list.assert_called_once_with( [ {
+  set_location_list_for_window.assert_called_once_with( 0, [ {
       'bufnr': 3,
       'lnum': 19,
       'col': 2,
@@ -641,3 +657,361 @@ def YouCompleteMe_UpdateDiagnosticInterface_PrioritizeErrorsOverWarnings_test(
       call( 'sign place 2 name=YcmWarning line=3 buffer=5' ),
       call( 'try | exec "sign unplace 1 buffer=5" | catch /E158/ | endtry' )
     ] )
+
+
+@YouCompleteMeInstance( { 'echo_current_diagnostic': 1,
+                          'always_populate_location_list': 1 } )
+@patch.object( ycm_buffer_module,
+               'DIAGNOSTIC_UI_ASYNC_FILETYPES',
+               [ 'ycmtest' ] )
+@patch( 'ycm.youcompleteme.YouCompleteMe.FiletypeCompleterExistsForFiletype',
+        return_value = True )
+@patch( 'ycm.vimsupport.PostVimMessage', new_callable = ExtendedMock )
+def YouCompleteMe_AsyncDiagnosticUpdate_SingleFile_test( ycm,
+                                                         post_vim_message,
+                                                         *args  ):
+
+  # This test simulates asynchronous diagnostic updates associated with a single
+  # file (e.g. Translation Unit), but where the actual errors refer to other
+  # open files and other non-open files. This is not strictly invalid, nor is it
+  # completely normal, but it is supported and does work.
+
+  # Contrast with the next test which sends the diagnostics filewise, which is
+  # what the language server protocol will do.
+
+  diagnostics = [
+    {
+      'kind': 'ERROR',
+      'text': 'error text in current buffer',
+      'location': {
+        'filepath': '/current',
+        'line_num': 1,
+        'column_num': 1
+      },
+    },
+    {
+      'kind': 'ERROR',
+      'text': 'error text in hidden buffer',
+      'location': {
+        'filepath': '/has_diags',
+        'line_num': 4,
+        'column_num': 2
+      },
+    },
+    {
+      'kind': 'ERROR',
+      'text': 'error text in buffer not open in Vim',
+      'location': {
+        'filepath': '/not_open',
+        'line_num': 8,
+        'column_num': 4
+      },
+    },
+  ]
+
+  current_buffer = VimBuffer( '/current',
+                              filetype = 'ycmtest',
+                              number = 1,
+                              window = 10 )
+  buffers = [
+    current_buffer,
+    VimBuffer( '/no_diags',
+               filetype = 'ycmtest',
+               number = 2,
+               window = 9 ),
+    VimBuffer( '/has_diags',
+               filetype = 'ycmtest',
+               number = 3,
+               window = 8 ),
+  ]
+
+  # Register each buffer internally with YCM
+  for current in buffers:
+    with MockVimBuffers( buffers, current, ( 1, 1 ) ):
+      ycm.OnFileReadyToParse()
+
+  with patch( 'ycm.vimsupport.SetLocationListForWindow',
+              new_callable = ExtendedMock ) as set_location_list_for_window:
+    with MockVimBuffers( buffers, current_buffer, ( 1, 1 ) ):
+      ycm.UpdateWithNewDiagnosticsForFile( '/current', diagnostics )
+
+    # We update the diagnostic on the current cursor position
+    post_vim_message.assert_has_exact_calls( [
+      call( "error text in current buffer", truncate = True, warning = False ),
+    ] )
+
+    # Ensure we included all the diags though
+    set_location_list_for_window.assert_has_exact_calls( [
+      call( 0, [
+        {
+          'lnum': 1,
+          'col': 1,
+          'bufnr': 1,
+          'valid': 1,
+          'type': 'E',
+          'text': 'error text in current buffer',
+        },
+        {
+          'lnum': 4,
+          'col': 2,
+          'bufnr': 3,
+          'valid': 1,
+          'type': 'E',
+          'text': 'error text in hidden buffer',
+        },
+        {
+          'lnum': 8,
+          'col': 4,
+          'bufnr': -1, # sic: Our mocked bufnr function actually returns -1,
+                       # even though YCM is passing "create if needed".
+                       # FIXME? we shouldn't do that, and we should pass
+                       # filename instead
+          'valid': 1,
+          'type': 'E',
+          'text': 'error text in buffer not open in Vim'
+        }
+      ] )
+    ] )
+
+
+@YouCompleteMeInstance( { 'echo_current_diagnostic': 1,
+                          'always_populate_location_list': 1 } )
+@patch.object( ycm_buffer_module,
+               'DIAGNOSTIC_UI_ASYNC_FILETYPES',
+               [ 'ycmtest' ] )
+@patch( 'ycm.youcompleteme.YouCompleteMe.FiletypeCompleterExistsForFiletype',
+        return_value = True )
+@patch( 'ycm.vimsupport.PostVimMessage', new_callable = ExtendedMock )
+def YouCompleteMe_AsyncDiagnosticUpdate_PerFile_test( ycm,
+                                                      post_vim_message,
+                                                      *args  ):
+
+  # This test simulates asynchronous diagnostic updates which are delivered per
+  # file, including files which are open and files which are not.
+
+  # Ordered to ensure that the calls to update are deterministic
+  diagnostics_per_file = [
+    ( '/current', [ {
+      'kind': 'ERROR',
+      'text': 'error text in current buffer',
+      'location': {
+        'filepath': '/current',
+        'line_num': 1,
+        'column_num': 1
+      }, }, ] ),
+    ( '/has_diags', [ {
+      'kind': 'ERROR',
+      'text': 'error text in hidden buffer',
+      'location': {
+        'filepath': '/has_diags',
+        'line_num': 4,
+        'column_num': 2
+      }, }, ] ),
+    ( '/not_open', [ {
+      'kind': 'ERROR',
+      'text': 'error text in buffer not open in Vim',
+      'location': {
+        'filepath': '/not_open',
+        'line_num': 8,
+        'column_num': 4
+      }, }, ] )
+  ]
+
+  current_buffer = VimBuffer( '/current',
+                              filetype = 'ycmtest',
+                              number = 1,
+                              window = 10 )
+  buffers = [
+    current_buffer,
+    VimBuffer( '/no_diags',
+               filetype = 'ycmtest',
+               number = 2,
+               window = 9 ),
+    VimBuffer( '/has_diags',
+               filetype = 'ycmtest',
+               number = 3,
+               window = 8 ),
+  ]
+
+  # Register each buffer internally with YCM
+  for current in buffers:
+    with MockVimBuffers( buffers, current, ( 1, 1 ) ):
+      ycm.OnFileReadyToParse()
+
+  with patch( 'ycm.vimsupport.SetLocationListForWindow',
+              new_callable = ExtendedMock ) as set_location_list_for_window:
+    with MockVimBuffers( buffers, current_buffer, ( 1, 1 ) ):
+      for filename, diagnostics in diagnostics_per_file:
+        ycm.UpdateWithNewDiagnosticsForFile( filename, diagnostics )
+
+    # We update the diagnostic on the current cursor position
+    post_vim_message.assert_has_exact_calls( [
+      call( "error text in current buffer", truncate = True, warning = False ),
+    ] )
+
+    # Ensure we included all the diags though
+    set_location_list_for_window.assert_has_exact_calls( [
+      call( 0, [
+        {
+          'lnum': 1,
+          'col': 1,
+          'bufnr': 1,
+          'valid': 1,
+          'type': 'E',
+          'text': 'error text in current buffer',
+        },
+      ] ),
+
+      call( 8, [
+        {
+          'lnum': 4,
+          'col': 2,
+          'bufnr': 3,
+          'valid': 1,
+          'type': 'E',
+          'text': 'error text in hidden buffer',
+        },
+      ] )
+    ] )
+
+
+@YouCompleteMeInstance()
+def YouCompleteMe_OnPeriodicTick_ServerNotRunning_test( ycm, *args  ):
+  with patch.object( ycm, 'IsServerAlive', return_value = False ):
+    assert_that( ycm.OnPeriodicTick(), equal_to( False ) )
+
+
+@YouCompleteMeInstance()
+def YouCompleteMe_OnPeriodicTick_ServerNotReady_test( ycm, *args ):
+  with patch.object( ycm, 'IsServerAlive', return_value = True ):
+    with patch.object( ycm, 'IsServerReady', return_value = False ):
+      assert_that( ycm.OnPeriodicTick(), equal_to( True ) )
+
+
+@YouCompleteMeInstance()
+@patch.object( ycm_buffer_module,
+               'DIAGNOSTIC_UI_ASYNC_FILETYPES',
+               [ 'ycmtest' ] )
+@patch( 'ycm.youcompleteme.YouCompleteMe.FiletypeCompleterExistsForFiletype',
+        return_value = True )
+@patch( 'ycm.client.base_request._ValidateResponseObject', return_value = True )
+@patch( 'ycm.client.base_request.BaseRequest.PostDataToHandlerAsync' )
+def YouCompelteMe_OnPeriodicTick_DontRetry_test( ycm,
+                                                 post_data_to_handler_async,
+                                                 *args ):
+
+  current_buffer = VimBuffer( '/current',
+                              filetype = 'ycmtest',
+                              number = 1,
+                              window = 10 )
+  buffers = [ current_buffer ]
+
+  # Create the request and make the first poll; we expect no response
+  with MockVimBuffers( buffers, current_buffer, ( 1, 1 ) ):
+      assert_that( ycm.OnPeriodicTick(), equal_to( True ) )
+      post_data_to_handler_async.assert_called()
+
+  assert ycm._message_poll_request is not None
+  post_data_to_handler_async.reset_mock()
+
+  # OK that sent the request, now poll to check if it is complete (say it is
+  # not)
+  with patch.object( ycm._message_poll_request,
+                     '_response_future',
+                     new = MockAsyncServerResponseInProgress() ) as mock_future:
+    poll_again = ycm.OnPeriodicTick()
+    mock_future.done.assert_called()
+    mock_future.result.assert_not_called()
+    assert_that( poll_again, equal_to( True ) )
+
+  # Poll again, but return a response (telling us to stop polling)
+  with patch.object( ycm._message_poll_request,
+                     '_response_future',
+                     new = MockAsyncServerResponseDone( False ) ) \
+      as mock_future:
+    poll_again = ycm.OnPeriodicTick()
+    mock_future.done.assert_called()
+    mock_future.result.assert_called()
+    post_data_to_handler_async.assert_not_called()
+    # We reset and don't poll anymore
+    assert_that( ycm._message_poll_request is None )
+    assert_that( poll_again, equal_to( False ) )
+
+
+
+@YouCompleteMeInstance()
+@patch.object( ycm_buffer_module,
+               'DIAGNOSTIC_UI_ASYNC_FILETYPES',
+               [ 'ycmtest' ] )
+@patch( 'ycm.youcompleteme.YouCompleteMe.FiletypeCompleterExistsForFiletype',
+        return_value = True )
+@patch( 'ycm.client.base_request._ValidateResponseObject', return_value = True )
+@patch( 'ycm.client.base_request.BaseRequest.PostDataToHandlerAsync' )
+def YouCompelteMe_OnPeriodicTick_Exception_test( ycm,
+                                                 post_data_to_handler_async,
+                                                 *args ):
+
+  current_buffer = VimBuffer( '/current',
+                              filetype = 'ycmtest',
+                              number = 1,
+                              window = 10 )
+  buffers = [ current_buffer ]
+
+  # Create the request and make the first poll; we expect no response
+  with MockVimBuffers( buffers, current_buffer, ( 1, 1 ) ):
+    assert_that( ycm.OnPeriodicTick(), equal_to( True ) )
+    post_data_to_handler_async.assert_called()
+
+  post_data_to_handler_async.reset_mock()
+
+  # Poll again, but return an exception response
+  mock_response = MockAsyncServerResponseException( RuntimeError( 'test' ) )
+  with patch.object( ycm._message_poll_request,
+                     '_response_future',
+                     new = mock_response ) as mock_future:
+    assert_that( ycm.OnPeriodicTick(), equal_to( False ) )
+    mock_future.done.assert_called()
+    mock_future.result.assert_called()
+    post_data_to_handler_async.assert_not_called()
+    # We reset and don't poll anymore
+    assert_that( ycm._message_poll_request is None )
+
+
+@YouCompleteMeInstance()
+@patch.object( ycm_buffer_module,
+               'DIAGNOSTIC_UI_ASYNC_FILETYPES',
+               [ 'ycmtest' ] )
+@patch( 'ycm.youcompleteme.YouCompleteMe.FiletypeCompleterExistsForFiletype',
+        return_value = True )
+@patch( 'ycm.client.base_request._ValidateResponseObject', return_value = True )
+@patch( 'ycm.client.base_request.BaseRequest.PostDataToHandlerAsync' )
+@patch( 'ycm.client.messages_request._HandlePollResponse' )
+def YouCompelteMe_OnPeriodicTick_ValidResponse_test( ycm,
+                                                     handle_poll_response,
+                                                     post_data_to_handler_async,
+                                                     *args ):
+
+  current_buffer = VimBuffer( '/current',
+                              filetype = 'ycmtest',
+                              number = 1,
+                              window = 10 )
+  buffers = [ current_buffer ]
+
+  # Create the request and make the first poll; we expect no response
+  with MockVimBuffers( buffers, current_buffer, ( 1, 1 ) ):
+    assert_that( ycm.OnPeriodicTick(), equal_to( True ) )
+    post_data_to_handler_async.assert_called()
+
+  post_data_to_handler_async.reset_mock()
+
+  # Poll again, and return a _proper_ response (finally!).
+  # Note, _HandlePollResponse is tested independently (for simplicity)
+  with patch.object( ycm._message_poll_request,
+                     '_response_future',
+                     new = MockAsyncServerResponseDone( [] ) ) as mock_future:
+    assert_that( ycm.OnPeriodicTick(), equal_to( True ) )
+    handle_poll_response.assert_called()
+    mock_future.done.assert_called()
+    mock_future.result.assert_called()
+    post_data_to_handler_async.assert_called() # Poll again!
+    assert_that( ycm._message_poll_request is not None )

python/ycm/vimsupport.py

@@ -235,9 +235,47 @@ def LineAndColumnNumbersClamped( line_num, column_num ):
 
 
 def SetLocationList( diagnostics ):
+  """Set the location list for the current window to the supplied diagnostics"""
+  SetLocationListForWindow( 0, diagnostics )
+
+
+def GetWindowNumberForBufferDiagnostics( buffer_number ):
+  """Return an appropriate window number to use for displaying diagnostics
+  associated with the buffer number supplied. Always returns a valid window
+  number or 0 meaning the current window."""
+
+  # Location lists are associated with _windows_ not _buffers_. This makes a lot
+  # of sense, but YCM associates diagnostics with _buffers_, because it is the
+  # buffer that actually gets parsed.
+  #
+  # The heuristic we use is to determine any open window for a specified buffer,
+  # and set that. If there is no such window on the current tab page, we use the
+  # current window (by passing 0 as the window number)
+
+  if buffer_number == vim.current.buffer.number:
+    return 0
+
+  window_number = GetIntValue( "bufwinnr({0})".format( buffer_number ) )
+
+  if window_number < 0:
+    return 0
+
+  return window_number
+
+
+def SetLocationListForBuffer( buffer_number, diagnostics ):
+  """Populate the location list of an apppropriate window for the supplied
+  buffer number. See SetLocationListForWindow for format of diagnostics."""
+  return SetLocationListForWindow(
+    GetWindowNumberForBufferDiagnostics( buffer_number ),
+    diagnostics )
+
+
+def SetLocationListForWindow( window_number, diagnostics ):
   """Populate the location list with diagnostics. Diagnostics should be in
   qflist format; see ":h setqflist" for details."""
-  vim.eval( 'setloclist( 0, {0} )'.format( json.dumps( diagnostics ) ) )
+  vim.eval( 'setloclist( {0}, {1} )'.format( window_number,
+                                             json.dumps( diagnostics ) ) )
 
 
 def OpenLocationList( focus = False, autoclose = False ):

python/ycm/youcompleteme.py

@@ -32,7 +32,9 @@ import vim
 from subprocess import PIPE
 from tempfile import NamedTemporaryFile
 from ycm import base, paths, vimsupport
-from ycm.buffer import BufferDict
+from ycm.buffer import ( BufferDict,
+                         DIAGNOSTIC_UI_FILETYPES,
+                         DIAGNOSTIC_UI_ASYNC_FILETYPES )
 from ycmd import utils
 from ycmd import server_utils
 from ycmd.request_wrap import RequestWrap
@@ -50,6 +52,7 @@ from ycm.client.debug_info_request import ( SendDebugInfoRequest,
 from ycm.client.omni_completion_request import OmniCompletionRequest
 from ycm.client.event_notification import SendEventNotificationAsync
 from ycm.client.shutdown_request import SendShutdownRequest
+from ycm.client.messages_request import MessagesPoll
 
 
 def PatchNoProxy():
@@ -97,8 +100,6 @@ CORE_OUTDATED_MESSAGE = (
   'YCM core library too old; PLEASE RECOMPILE by running the install.py '
   'script. See the documentation for more details.' )
 SERVER_IDLE_SUICIDE_SECONDS = 1800  # 30 minutes
-DIAGNOSTIC_UI_FILETYPES = set( [ 'cpp', 'cs', 'c', 'objc', 'objcpp',
-                                 'typescript' ] )
 CLIENT_LOGFILE_FORMAT = 'ycm_'
 SERVER_LOGFILE_FORMAT = 'ycmd_{port}_{std}_'
 
@@ -123,19 +124,20 @@ class YouCompleteMe( object ):
     self._filetypes_with_keywords_loaded = set()
     self._ycmd_keepalive = YcmdKeepalive()
     self._server_is_ready_with_cache = False
-    self._SetupLogging()
+    self._SetUpLogging()
-    self._SetupServer()
+    self._SetUpServer()
     self._ycmd_keepalive.Start()
     self._complete_done_hooks = {
       'cs': lambda self: self._OnCompleteDone_Csharp()
     }
 
 
-  def _SetupServer( self ):
+  def _SetUpServer( self ):
     self._available_completers = {}
     self._user_notified_about_crash = False
     self._filetypes_with_keywords_loaded = set()
     self._server_is_ready_with_cache = False
+    self._message_poll_request = None
 
     hmac_secret = os.urandom( HMAC_SECRET_LENGTH )
     options_dict = dict( self._user_options )
@@ -186,7 +188,7 @@ class YouCompleteMe( object ):
                                           stdout = PIPE, stderr = PIPE )
 
 
-  def _SetupLogging( self ):
+  def _SetUpLogging( self ):
     def FreeFileFromOtherProcesses( file_object ):
       if utils.OnWindows():
         from ctypes import windll
@@ -283,7 +285,7 @@ class YouCompleteMe( object ):
   def RestartServer( self ):
     vimsupport.PostVimMessage( 'Restarting ycmd server...' )
     self._ShutdownServer()
-    self._SetupServer()
+    self._SetUpServer()
 
 
   def SendCompletionRequest( self, force_semantic = False ):
@@ -364,6 +366,59 @@ class YouCompleteMe( object ):
     return self.CurrentBuffer().NeedsReparse()
 
 
+  def UpdateWithNewDiagnosticsForFile( self, filepath, diagnostics ):
+    bufnr = vimsupport.GetBufferNumberForFilename( filepath )
+    if bufnr in self._buffers and vimsupport.BufferIsVisible( bufnr ):
+      # Note: We only update location lists, etc. for visible buffers, because
+      # otherwise we defualt to using the curren location list and the results
+      # are that non-visible buffer errors clobber visible ones.
+      self._buffers[ bufnr ].UpdateWithNewDiagnostics( diagnostics )
+    else:
+      # The project contains errors in file "filepath", but that file is not
+      # open in any buffer. This happens for Language Server Protocol-based
+      # completers, as they return diagnostics for the entire "project"
+      # asynchronously (rather than per-file in the response to the parse
+      # request).
+      #
+      # There are a number of possible approaches for
+      # this, but for now we simply ignore them. Other options include:
+      # - Use the QuickFix list to report project errors?
+      # - Use a special buffer for project errors
+      # - Put them in the location list of whatever the "current" buffer is
+      # - Store them in case the buffer is opened later
+      # - add a :YcmProjectDiags command
+      # - Add them to errror/warning _counts_ but not any actual location list
+      #   or other
+      # - etc.
+      #
+      # However, none of those options are great, and lead to their own
+      # complexities. So for now, we just ignore these diagnostics for files not
+      # open in any buffer.
+      pass
+
+
+  def OnPeriodicTick( self ):
+    if not self.IsServerAlive():
+      # Server has died. We'll reset when the server is started again.
+      return False
+    elif not self.IsServerReady():
+      # Try again in a jiffy
+      return True
+
+    if not self._message_poll_request:
+      self._message_poll_request = MessagesPoll()
+
+    if not self._message_poll_request.Poll( self ):
+      # Don't poll again until some event which might change the server's mind
+      # about whether to provide messages for the current buffer (e.g. buffer
+      # visit, file ready to parse, etc.)
+      self._message_poll_request = None
+      return False
+
+    # Poll again in a jiffy
+    return True
+
+
   def OnFileReadyToParse( self ):
     if not self.IsServerAlive():
       self.NotifyUserIfServerCrashed()
@@ -534,7 +589,8 @@ class YouCompleteMe( object ):
 
 
   def DiagnosticUiSupportedForCurrentFiletype( self ):
-    return any( [ x in DIAGNOSTIC_UI_FILETYPES
+    return any( [ x in DIAGNOSTIC_UI_FILETYPES or
+                  x in DIAGNOSTIC_UI_ASYNC_FILETYPES
                   for x in vimsupport.CurrentFiletypes() ] )
 
 
@@ -566,7 +622,9 @@ class YouCompleteMe( object ):
          self.NativeFiletypeCompletionUsable() ):
 
       if self.ShouldDisplayDiagnostics():
-        current_buffer.UpdateDiagnostics()
+        # Forcefuly update the location list, etc. from the parse request when
+        # doing something like :YcmDiags
+        current_buffer.UpdateDiagnostics( block is True )
       else:
         # YCM client has a hard-coded list of filetypes which are known
         # to support diagnostics, self.DiagnosticUiSupportedForCurrentFiletype()
@@ -683,7 +741,7 @@ class YouCompleteMe( object ):
     if '*' in filetype_to_disable:
       return False
     else:
-      return not any([ x in filetype_to_disable for x in filetypes ])
+      return not any( [ x in filetype_to_disable for x in filetypes ] )
 
 
   def ShowDetailedDiagnostic( self ):