Merge pull request #578 from Valloric/ycmd

YCM rewritten to have a client-server architecture

.gitmodules

@@ -1,6 +1,24 @@
-[submodule "python/ycm/completers/python/jedi"]
+[submodule "third_party/jedi"]
-	path = python/ycm/completers/python/jedi
+	path = third_party/jedi
-	url = https://github.com/davidhalter/jedi.git
+	url = https://github.com/davidhalter/jedi
 [submodule "python/ycm/completers/cs/OmniSharpServer"]
 	path = python/ycm/completers/cs/OmniSharpServer
 	url = https://github.com/nosami/OmniSharpServer.git
+[submodule "third_party/requests-futures"]
+	path = third_party/requests-futures
+	url = https://github.com/ross/requests-futures
+[submodule "third_party/requests"]
+	path = third_party/requests
+	url = https://github.com/kennethreitz/requests
+[submodule "third_party/waitress"]
+	path = third_party/waitress
+	url = https://github.com/Pylons/waitress
+[submodule "third_party/bottle"]
+	path = third_party/bottle
+	url = https://github.com/defnull/bottle
+[submodule "third_party/frozendict"]
+	path = third_party/frozendict
+	url = https://github.com/slezica/python-frozendict
+[submodule "third_party/argparse"]
+	path = third_party/argparse
+	url = https://github.com/bewest/argparse

.travis.yml

@@ -4,10 +4,11 @@ python:
   - "2.7"
 install:
   - pip install -r python/test_requirements.txt --use-mirrors
+  - sudo apt-get install mono-devel
 compiler:
   - gcc
   - clang
-script: flake8 --exclude=jedi --select=F,C9 --max-complexity=10 python && ./install.sh && nosetests python
+script: ./run_tests.sh
 env:
-  - YCM_TESTRUN=1 EXTRA_CMAKE_ARGS=""
+  - USE_CLANG_COMPLETER="true"
-  - YCM_TESTRUN=1 EXTRA_CMAKE_ARGS="-DUSE_CLANG_COMPLETER=ON"
+  - USE_CLANG_COMPLETER="false"

CONTRIBUTING.md

@@ -44,14 +44,22 @@ Here are the things you should do when creating an issue:
 1. **Write a step-by-step procedure that when performed repeatedly reproduces
    your issue.** If we can't reproduce the issue, then we can't fix it. It's
    that simple.
-2. **Create a test case for your issue**. This is critical. Don't talk about how
+2. Put the following options in your vimrc:
+   ```
+   let g:ycm_server_use_vim_stdout = 1
+   let g:ycm_server_log_level = 'debug'
+   ```
+   Then start gvim/macvim (not console vim) from the console. As you use Vim,
+   you'll see the `ycmd` debug output stream in the console. Attach that to you
+   issue.
+3. **Create a test case for your issue**. This is critical. Don't talk about how
    "when I have X in my file" or similar, _create a file with X in it_ and put
    the contents inside code blocks in your issue description. Try to make this
    test file _as small as possible_. Don't just paste a huge, 500 line source
    file you were editing and present that as a test. _Minimize_ the file so that
    the problem is reproduced with the smallest possible amount of test data.
-3. **Include your OS and OS version.**
+4. **Include your OS and OS version.**
-4. **Include the output of `vim --version`.**
+5. **Include the output of `vim --version`.**
 
 
 Creating good pull requests

README.md

@@ -89,8 +89,9 @@ local binary folder (for example `/usr/local/bin/mvim`) and then symlink it:
 Install YouCompleteMe with [Vundle][].
 
 **Remember:** YCM is a plugin with a compiled component. If you **update** YCM
-using Vundle and the ycm_core library API has changed (happens rarely), YCM will
+using Vundle and the ycm_support_libs library APIs have changed (happens
-notify you to recompile it. You should then rerun the install process.
+rarely), YCM will notify you to recompile it. You should then rerun the install
+process.
 
 It's recommended that you have the latest Xcode installed along with the latest
 Command Line Tools (that you install from within Xcode).
@@ -136,8 +137,9 @@ from source][vim-build] (don't worry, it's easy).
 Install YouCompleteMe with [Vundle][].
 
 **Remember:** YCM is a plugin with a compiled component. If you **update** YCM
-using Vundle and the ycm_core library API has changed (happens rarely), YCM will
+using Vundle and the ycm_support_libs library APIs have changed (happens
-notify you to recompile it. You should then rerun the install process.
+rarely), YCM will notify you to recompile it. You should then rerun the install
+process.
 
 Install development tools and CMake: `sudo apt-get install build-essential cmake`
 
@@ -184,8 +186,9 @@ that platform).
 See the _FAQ_ if you have any issues.
 
 **Remember:** YCM is a plugin with a compiled component. If you **update** YCM
-using Vundle and the ycm_core library API has changed (happens rarely), YCM will
+using Vundle and the ycm_support_libs library APIs have changed (happens
-notify you to recompile it. You should then rerun the install process.
+rarely), YCM will notify you to recompile it. You should then rerun the install
+process.
 
 **Please follow the instructions carefully. Read EVERY WORD.**
 
@@ -221,8 +224,8 @@ notify you to recompile it. You should then rerun the install process.
     binaries from llvm.org][clang-download] if at all possible. Make sure you
     download the correct archive file for your OS.
 
-4.  **Compile the `ycm_core` plugin plugin** (ha!) that YCM needs. This is the
+4.  **Compile the `ycm_support_libs` libraries** that YCM needs. These libs
-    C++ engine that YCM uses to get fast completions.
+    are the C++ engines that YCM uses to get fast completions.
 
     You will need to have `cmake` installed in order to generate the required
     makefiles. Linux users can install cmake with their package manager (`sudo
@@ -261,7 +264,7 @@ notify you to recompile it. You should then rerun the install process.
 
     Now that makefiles have been generated, simply run:
 
-        make ycm_core
+        make ycm_support_libs
 
     For those who want to use the system version of libclang, you would pass
     `-DUSE_SYSTEM_LIBCLANG=ON` to cmake _instead of_ the
@@ -323,6 +326,13 @@ YCM automatically detects which completion engine would be the best in any
 situation. On occasion, it queries several of them at once, merges the
 outputs and presents the results to you.
 
+### Client-server architecture
+
+YCM has a client-server architecture; the Vim part of YCM is only a thin client
+that talks to the `ycmd` HTTP+JSON server that has the vast majority of YCM
+logic and functionality. The server is started and stopped automatically as you
+start and stop Vim.
+
 ### Completion string ranking
 
 The subsequence filter removes any completions that do not match the input, but
@@ -498,6 +508,11 @@ yours truly.
 Commands
 --------
 
+### The `:YcmRestartServer` command
+
+If the `ycmd` completion server suddenly stops for some reason, you can restart
+it with this command.
+
 ### The `:YcmForceCompileAndDiagnostics` command
 
 Calling this command will force YCM to immediately recompile your file
@@ -832,6 +847,64 @@ Default: `0`
 
     let g:ycm_seed_identifiers_with_syntax = 0
 
+### The `g:ycm_server_use_vim_stdout` option
+
+By default, the `ycmd` completion server writes logs to logfiles. When this
+option is set to `1`, the server writes logs to Vim's stdout (so you'll see them
+in the console).
+
+Default: `0`
+
+    let g:ycm_server_use_vim_stdout = 0
+
+### The `g:ycm_server_keep_logfiles` option
+
+When this option is set to `1`, the `ycmd` completion server will keep the
+logfiles around after shutting down (they are deleted on shutdown by default).
+
+To see where the logfiles are, call `:YcmDebugInfo`.
+
+Default: `0`
+
+    let g:ycm_server_keep_logfiles = 0
+
+### The `g:ycm_server_log_level` option
+
+The logging level that the `ycmd` completion server uses. Valid values are the
+following, from most verbose to least verbose:
+- `debug`
+- `info`
+- `warning`
+- `error`
+- `critical`
+
+Note that `debug` is _very_ verbose.
+
+Default: `info`
+
+    let g:ycm_server_log_level = 'info'
+
+### The `g:ycm_server_idle_suicide_seconds` option
+
+This option sets the number of seconds of `ycmd` server idleness (no requests
+received) after which the server stops itself. NOTE: the YCM Vim client sends a
+shutdown request to the server when Vim is shutting down.
+
+If your Vim crashes for instance, `ycmd` never gets the shutdown command and
+becomes a zombie process. This option prevents such zombies from sticking around
+forever.
+
+The default option is `43200` seconds which is 12 hours. The reason for the
+interval being this long is to prevent the server from shutting down if you
+leave your computer (and Vim) turned on during the night.
+
+The server "heartbeat" that checks whether this interval has passed occurs every
+10 minutes.
+
+Default: `43200`
+
+    let g:ycm_server_idle_suicide_seconds = 43200
+
 ### The `g:ycm_csharp_server_port` option
 
 The port number (on `localhost`) on which the OmniSharp server should be

autoload/youcompleteme.vim

@@ -22,8 +22,6 @@ set cpo&vim
 " This needs to be called outside of a function
 let s:script_folder_path = escape( expand( '<sfile>:p:h' ), '\' )
 let s:searched_and_results_found = 0
-let s:should_use_filetype_completion = 0
-let s:completion_start_column = 0
 let s:omnifunc_mode = 0
 
 let s:old_cursor_position = []
@@ -31,28 +29,54 @@ let s:cursor_moved = 0
 let s:moved_vertically_in_insert_mode = 0
 let s:previous_num_chars_on_current_line = -1
 
+let s:forced_syntastic_checker_for = {
+      \ 'cpp': 1,
+      \ 'c': 1,
+      \ 'objc': 1,
+      \ 'objcpp': 1,
+      \ }
+
+
 function! youcompleteme#Enable()
   " When vim is in diff mode, don't run
   if &diff
     return
   endif
 
+  call s:SetUpBackwardsCompatibility()
+
   py import sys
   py import vim
   exe 'python sys.path.insert( 0, "' . s:script_folder_path . '/../python" )'
-  py from ycm import extra_conf_store
+  py from ycm import utils
-  py extra_conf_store.CallExtraConfYcmCorePreloadIfExists()
+  py utils.AddThirdPartyFoldersToSysPath()
   py from ycm import base
+  py base.LoadJsonDefaultsIntoVim()
+  py from ycm import vimsupport
+  py from ycm import user_options_store
+  py user_options_store.SetAll( base.BuildServerConf() )
 
   if !pyeval( 'base.CompatibleWithYcmCore()')
     echohl WarningMsg |
-      \ echomsg "YouCompleteMe unavailable: ycm_core too old, PLEASE RECOMPILE ycm_core" |
+      \ echomsg "YouCompleteMe unavailable: YCM support libs too old, PLEASE RECOMPILE" |
       \ echohl None
     return
   endif
 
   py from ycm.youcompleteme import YouCompleteMe
-  py ycm_state = YouCompleteMe()
+  py ycm_state = YouCompleteMe( user_options_store.GetAll() )
+
+  call s:SetUpCpoptions()
+  call s:SetUpCompleteopt()
+  call s:SetUpKeyMappings()
+
+  if g:ycm_register_as_syntastic_checker
+    call s:ForceSyntasticCFamilyChecker()
+  endif
+
+  if g:ycm_allow_changing_updatetime
+    set ut=2000
+  endif
 
   augroup youcompleteme
     autocmd!
@@ -63,7 +87,10 @@ function! youcompleteme#Enable()
     " is read. This is because youcompleteme#Enable() is called on VimEnter and
     " that happens *after" BufRead/BufEnter has already triggered for the
     " initial file.
-    autocmd BufRead,BufEnter * call s:OnBufferVisit()
+    " We also need to trigger buf init code on the FileType event because when
+    " the user does :enew and then :set ft=something, we need to run buf init
+    " code again.
+    autocmd BufRead,BufEnter,FileType * call s:OnBufferVisit()
     autocmd BufUnload * call s:OnBufferUnload( expand( '<afile>:p' ) )
     autocmd CursorHold,CursorHoldI * call s:OnCursorHold()
     autocmd InsertLeave * call s:OnInsertLeave()
@@ -71,19 +98,6 @@ function! youcompleteme#Enable()
     autocmd VimLeave * call s:OnVimLeave()
   augroup END
 
-  call s:SetUpCpoptions()
-  call s:SetUpCompleteopt()
-  call s:SetUpKeyMappings()
-  call s:SetUpBackwardsCompatibility()
-
-  if g:ycm_register_as_syntastic_checker
-    call s:ForceSyntasticCFamilyChecker()
-  endif
-
-  if g:ycm_allow_changing_updatetime
-    set ut=2000
-  endif
-
   " Calling this once solves the problem of BufRead/BufEnter not triggering for
   " the first loaded file. This should be the last command executed in this
   " function!
@@ -151,6 +165,11 @@ function! s:SetUpBackwardsCompatibility()
     let g:ycm_complete_in_strings = 1
     let g:ycm_complete_in_comments = 1
   endif
+
+  " ycm_filetypes_to_completely_ignore is the old name for fileype_blacklist
+  if has_key( g:, 'ycm_filetypes_to_completely_ignore' )
+    let g:filetype_blacklist =  g:ycm_filetypes_to_completely_ignore
+  endif
 endfunction
 
 
@@ -163,6 +182,12 @@ function! s:ForceSyntasticCFamilyChecker()
 endfunction
 
 
+function! s:ForcedAsSyntasticCheckerForCurrentFiletype()
+  return g:ycm_register_as_syntastic_checker &&
+         \ get( s:forced_syntastic_checker_for, &filetype, 0 )
+endfunction
+
+
 function! s:AllowedToCompleteInCurrentFile()
   if empty( &filetype ) || getbufvar(winbufnr(winnr()), "&buftype") ==# 'nofile'
     return 0
@@ -221,7 +246,6 @@ endfunction
 
 function! s:OnVimLeave()
   py ycm_state.OnVimLeave()
-  py extra_conf_store.CallExtraConfVimCloseIfExists()
 endfunction
 
 
@@ -257,9 +281,6 @@ function! s:OnCursorHold()
   endif
 
   call s:SetUpCompleteopt()
-  " Order is important here; we need to extract any done diagnostics before
-  " reparsing the file again
-  call s:UpdateDiagnosticNotifications()
   call s:OnFileReadyToParse()
 endfunction
 
@@ -269,6 +290,12 @@ function! s:OnFileReadyToParse()
   " happen for special buffers.
   call s:SetUpYcmChangedTick()
 
+  " Order is important here; we need to extract any done diagnostics before
+  " reparsing the file again. If we sent the new parse request first, then
+  " the response would always be pending when we called
+  " UpdateDiagnosticNotifications.
+  call s:UpdateDiagnosticNotifications()
+
   let buffer_changed = b:changedtick != b:ycm_changedtick.file_ready_to_parse
   if buffer_changed
     py ycm_state.OnFileReadyToParse()
@@ -326,7 +353,6 @@ function! s:OnCursorMovedNormalMode()
     return
   endif
 
-  call s:UpdateDiagnosticNotifications()
   call s:OnFileReadyToParse()
 endfunction
 
@@ -337,7 +363,6 @@ function! s:OnInsertLeave()
   endif
 
   let s:omnifunc_mode = 0
-  call s:UpdateDiagnosticNotifications()
   call s:OnFileReadyToParse()
   py ycm_state.OnInsertLeave()
   if g:ycm_autoclose_preview_window_after_completion ||
@@ -407,10 +432,16 @@ endfunction
 
 
 function! s:UpdateDiagnosticNotifications()
-  if get( g:, 'loaded_syntastic_plugin', 0 ) &&
+  let should_display_diagnostics =
-        \ pyeval( 'ycm_state.NativeFiletypeCompletionUsable()' ) &&
+        \ get( g:, 'loaded_syntastic_plugin', 0 ) &&
-        \ pyeval( 'ycm_state.DiagnosticsForCurrentFileReady()' ) &&
+        \ s:ForcedAsSyntasticCheckerForCurrentFiletype() &&
-        \ g:ycm_register_as_syntastic_checker
+        \ pyeval( 'ycm_state.NativeFiletypeCompletionUsable()' )
+
+  if !should_display_diagnostics
+    return
+  endif
+
+  if pyeval( 'ycm_state.DiagnosticsForCurrentFileReady()' )
     SyntasticCheck
   endif
 endfunction
@@ -494,29 +525,24 @@ function! s:InvokeCompletion()
 endfunction
 
 
-function! s:CompletionsForQuery( query, use_filetype_completer,
+python << EOF
-      \ completion_start_column )
+def GetCompletions( query ):
-  if a:use_filetype_completer
+  request = ycm_state.GetCurrentCompletionRequest()
-    py completer = ycm_state.GetFiletypeCompleter()
+  request.Start( query )
-  else
+  while not request.Done():
-    py completer = ycm_state.GetGeneralCompleter()
+    if bool( int( vim.eval( 'complete_check()' ) ) ):
-  endif
-
-  py completer.CandidatesForQueryAsync( vim.eval( 'a:query' ),
-        \ int( vim.eval( 'a:completion_start_column' ) ) )
-
-  let l:results_ready = 0
-  while !l:results_ready
-    let l:results_ready = pyeval( 'completer.AsyncCandidateRequestReady()' )
-    if complete_check()
-      let s:searched_and_results_found = 0
       return { 'words' : [], 'refresh' : 'always'}
-    endif
-  endwhile
 
-  let l:results = pyeval( 'base.AdjustCandidateInsertionText( completer.CandidatesFromStoredRequest() )' )
+  results = base.AdjustCandidateInsertionText( request.Response() )
-  let s:searched_and_results_found = len( l:results ) != 0
+  return { 'words' : results, 'refresh' : 'always' }
-  return { 'words' : l:results, 'refresh' : 'always' }
+EOF
+
+
+function! s:CompletionsForQuery( query )
+  py results = GetCompletions( vim.eval( 'a:query' ) )
+  let results = pyeval( 'results' )
+  let s:searched_and_results_found = len( results.words ) != 0
+  return results
 endfunction
 
 
@@ -540,24 +566,13 @@ function! youcompleteme#Complete( findstart, base )
       return -2
     endif
 
-
+    py request = ycm_state.CreateCompletionRequest()
-    " TODO: make this a function-local variable instead of a script-local one
+    if !pyeval( 'bool(request)' )
-    let s:completion_start_column = pyeval( 'base.CompletionStartColumn()' )
-    let s:should_use_filetype_completion =
-          \ pyeval( 'ycm_state.ShouldUseFiletypeCompleter(' .
-          \ s:completion_start_column . ')' )
-
-    if !s:should_use_filetype_completion &&
-          \ !pyeval( 'ycm_state.ShouldUseGeneralCompleter(' .
-          \ s:completion_start_column . ')' )
-      " for vim, -2 means not found but don't trigger an error message
-      " see :h complete-functions
       return -2
     endif
-    return s:completion_start_column
+    return pyeval( 'request.CompletionStartColumn()' )
   else
-    return s:CompletionsForQuery( a:base, s:should_use_filetype_completion,
+    return s:CompletionsForQuery( a:base )
-          \ s:completion_start_column )
   endif
 endfunction
 
@@ -565,14 +580,26 @@ endfunction
 function! youcompleteme#OmniComplete( findstart, base )
   if a:findstart
     let s:omnifunc_mode = 1
-    let s:completion_start_column = pyeval( 'base.CompletionStartColumn()' )
+    py request = ycm_state.CreateCompletionRequest( force_semantic = True )
-    return s:completion_start_column
+    return pyeval( 'request.CompletionStartColumn()' )
   else
-    return s:CompletionsForQuery( a:base, 1, s:completion_start_column )
+    return s:CompletionsForQuery( a:base )
   endif
 endfunction
 
 
+function! youcompleteme#ServerPid()
+  return pyeval( 'ycm_state.ServerPid()' )
+endfunction
+
+
+function! s:RestartServer()
+  py ycm_state.RestartServer()
+endfunction
+
+command! YcmRestartServer call s:RestartServer()
+
+
 function! s:ShowDetailedDiagnostic()
   py ycm_state.ShowDetailedDiagnostic()
 endfunction
@@ -584,7 +611,7 @@ command! YcmShowDetailedDiagnostic call s:ShowDetailedDiagnostic()
 " required (currently that's on buffer save) OR when the SyntasticCheck command
 " is invoked
 function! youcompleteme#CurrentFileDiagnostics()
-  return pyeval( 'ycm_state.GetDiagnosticsForCurrentFile()' )
+  return pyeval( 'ycm_state.GetDiagnosticsFromStoredRequest()' )
 endfunction
 
 
@@ -598,37 +625,27 @@ endfunction
 
 command! YcmDebugInfo call s:DebugInfo()
 
+
 function! s:CompleterCommand(...)
   " CompleterCommand will call the OnUserCommand function of a completer.
   " If the first arguments is of the form "ft=..." it can be used to specify the
   " completer to use (for example "ft=cpp").  Else the native filetype completer
   " of the current buffer is used.  If no native filetype completer is found and
-  " no completer was specified this throws an error.  You can use "ft=ycm:omni"
+  " no completer was specified this throws an error.  You can use
-  " to select the omni completer or "ft=ycm:ident" to select the identifier
+  " "ft=ycm:ident" to select the identifier completer.
-  " completer.  The remaining arguments will passed to the completer.
+  " The remaining arguments will be passed to the completer.
   let arguments = copy(a:000)
+  let completer = ''
 
   if a:0 > 0 && strpart(a:1, 0, 3) == 'ft='
-    if a:1 == 'ft=ycm:omni'
+    if a:1 == 'ft=ycm:ident'
-      py completer = ycm_state.GetOmniCompleter()
+      let completer = 'identifier'
-    elseif a:1 == 'ft=ycm:ident'
-      py completer = ycm_state.GetGeneralCompleter()
-    else
-      py completer = ycm_state.GetFiletypeCompleterForFiletype(
-                   \ vim.eval('a:1').lstrip('ft=') )
     endif
     let arguments = arguments[1:]
-  elseif pyeval( 'ycm_state.NativeFiletypeCompletionAvailable()' )
-    py completer = ycm_state.GetFiletypeCompleter()
-  else
-    echohl WarningMsg |
-      \ echomsg "No native completer found for current buffer." |
-      \ echomsg  "Use ft=... as the first argument to specify a completer." |
-      \ echohl None
-    return
   endif
 
-  py completer.OnUserCommand( vim.eval( 'l:arguments' ) )
+  py ycm_state.SendCommandRequest( vim.eval( 'l:arguments' ),
+        \                          vim.eval( 'l:completer' ) )
 endfunction
 
 
@@ -645,9 +662,8 @@ endfunction
 command! -nargs=* -complete=custom,youcompleteme#SubCommandsComplete
   \ YcmCompleter call s:CompleterCommand(<f-args>)
 
-
 function! youcompleteme#SubCommandsComplete( arglead, cmdline, cursorpos )
-  return join( pyeval( 'ycm_state.GetFiletypeCompleter().DefinedSubcommands()' ),
+  return join( pyeval( 'ycm_state.GetDefinedSubcommands()' ),
     \ "\n")
 endfunction
 
@@ -668,14 +684,6 @@ function! s:ForceCompile()
       break
     endif
 
-    let getting_completions = pyeval(
-          \ 'ycm_state.GettingCompletions()' )
-
-    if !getting_completions
-      echom "Unable to retrieve diagnostics, see output of `:mes` for possible details."
-      return 0
-    endif
-
     sleep 100m
   endwhile
   return 1
@@ -701,7 +709,7 @@ function! s:ShowDiagnostics()
     return
   endif
 
-  let diags = pyeval( 'ycm_state.GetDiagnosticsForCurrentFile()' )
+  let diags = pyeval( 'ycm_state.GetDiagnosticsFromStoredRequest()' )
   if !empty( diags )
     call setloclist( 0, diags )
     lopen

cpp/BoostParts/CMakeLists.txt

@@ -26,8 +26,8 @@ cmake_minimum_required( VERSION 2.8 )
 
 project( BoostParts )
 
-set( Python_ADDITIONAL_VERSIONS 2.7 2.6 2.5 )
+set( Python_ADDITIONAL_VERSIONS 2.7 2.6 )
-find_package( PythonLibs 2.5 REQUIRED )
+find_package( PythonLibs 2.6 REQUIRED )
 
 if ( NOT PYTHONLIBS_VERSION_STRING VERSION_LESS "3.0.0" )
   message( FATAL_ERROR

cpp/CMakeLists.txt

@@ -138,19 +138,18 @@ endif()
 # the compiler to output a warning during linking:
 #  clang: warning: argument unused during compilation: '-std=c++0x'
 # This is caused by cmake passing this flag to the linking stage which it
-# shouldn't do. It's ignored so it does no harm, but the warning is annoying and
+# shouldn't do. It's ignored so it does no harm, but the warning is annoying.
-# there's no way around the problem (the flag is correctly used during the
+#
-# compilation stage). We could use add_definitions(-std=c++0x), but this will
+# Putting the flag in add_definitions() works around the issue, even though it
-# break the llvm build since the flag will then be used when compiling C code
+# shouldn't in theory go there.
-# too. Sadly there's no way around the warning.
 if ( CPP11_AVAILABLE )
   message( "Your C++ compiler supports C++11, compiling in that mode." )
 
   # Cygwin needs its hand held a bit; see issue #473
   if ( CYGWIN AND CMAKE_COMPILER_IS_GNUCXX )
-    set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++0x" )
+    add_definitions( -std=gnu++0x )
   else()
-    set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x" )
+    add_definitions( -std=c++0x )
   endif()
 else()
   message(

cpp/ycm/CMakeLists.txt

@@ -17,10 +17,12 @@
 
 cmake_minimum_required( VERSION 2.8 )
 
-project( ycm_core )
+project( ycm_support_libs )
+set( CLIENT_LIB "ycm_client_support" )
+set( SERVER_LIB "ycm_core" )
 
-set( Python_ADDITIONAL_VERSIONS 2.7 2.6 2.5 )
+set( Python_ADDITIONAL_VERSIONS 2.7 2.6 )
-find_package( PythonLibs 2.5 REQUIRED )
+find_package( PythonLibs 2.6 REQUIRED )
 
 if ( NOT PYTHONLIBS_VERSION_STRING VERSION_LESS "3.0.0" )
   message( FATAL_ERROR
@@ -138,15 +140,15 @@ include_directories(
   ${CLANG_INCLUDES_DIR}
   )
 
-file( GLOB_RECURSE SOURCES *.h *.cpp )
+file( GLOB_RECURSE SERVER_SOURCES *.h *.cpp )
 
 # The test sources are a part of a different target, so we remove them
 # The CMakeFiles cpp file is picked up when the user creates an in-source build,
-# and we don't want that.
+# and we don't want that. We also remove client-specific code
-file( GLOB_RECURSE to_remove tests/*.h tests/*.cpp CMakeFiles/*.cpp )
+file( GLOB_RECURSE to_remove tests/*.h tests/*.cpp CMakeFiles/*.cpp *client* )
 
 if( to_remove )
-  list( REMOVE_ITEM SOURCES ${to_remove} )
+  list( REMOVE_ITEM SERVER_SOURCES ${to_remove} )
 endif()
 
 if ( USE_CLANG_COMPLETER )
@@ -158,7 +160,7 @@ else()
   file( GLOB_RECURSE to_remove_clang ClangCompleter/*.h ClangCompleter/*.cpp )
 
   if( to_remove_clang )
-    list( REMOVE_ITEM SOURCES ${to_remove_clang} )
+    list( REMOVE_ITEM SERVER_SOURCES ${to_remove_clang} )
   endif()
 endif()
 
@@ -217,11 +219,33 @@ endif()
 
 #############################################################################
 
-add_library( ${PROJECT_NAME} SHARED
+# We don't actually need all of the files this picks up, just the ones needed by
-             ${SOURCES}
+# PythonSupport.cpp. But this is easier to maintain and dead code elemination
+# will remove unused code.
+file( GLOB CLIENT_SOURCES *.h *.cpp )
+file( GLOB SERVER_SPECIFIC *ycm_core* )
+
+if( SERVER_SPECIFIC )
+  list( REMOVE_ITEM CLIENT_SOURCES ${SERVER_SPECIFIC} )
+endif()
+
+add_library( ${CLIENT_LIB} SHARED
+             ${CLIENT_SOURCES}
            )
 
-target_link_libraries( ${PROJECT_NAME}
+target_link_libraries( ${CLIENT_LIB}
+                       BoostParts
+                       ${PYTHON_LIBRARIES}
+                       ${EXTRA_LIBS}
+                     )
+
+#############################################################################
+
+add_library( ${SERVER_LIB} SHARED
+             ${SERVER_SOURCES}
+           )
+
+target_link_libraries( ${SERVER_LIB}
                        BoostParts
                        ${PYTHON_LIBRARIES}
                        ${LIBCLANG_TARGET}
@@ -231,35 +255,43 @@ target_link_libraries( ${PROJECT_NAME}
 if( LIBCLANG_TARGET )
   if( NOT WIN32 )
     add_custom_command(
-      TARGET ${PROJECT_NAME}
+      TARGET ${SERVER_LIB}
       POST_BUILD
-      COMMAND ${CMAKE_COMMAND} -E copy "${LIBCLANG_TARGET}" "$<TARGET_FILE_DIR:${PROJECT_NAME}>"
+      COMMAND ${CMAKE_COMMAND} -E copy "${LIBCLANG_TARGET}" "$<TARGET_FILE_DIR:${SERVER_LIB}>"
     )
   else()
     add_custom_command(
-      TARGET ${PROJECT_NAME}
+      TARGET ${SERVER_LIB}
       POST_BUILD
-      COMMAND ${CMAKE_COMMAND} -E copy "${PATH_TO_LLVM_ROOT}/bin/libclang.dll" "$<TARGET_FILE_DIR:${PROJECT_NAME}>")
+      COMMAND ${CMAKE_COMMAND} -E copy "${PATH_TO_LLVM_ROOT}/bin/libclang.dll" "$<TARGET_FILE_DIR:${SERVER_LIB}>")
   endif()
 endif()
 
+#############################################################################
+
+# Convenience target that builds both support libs.
+add_custom_target( ${PROJECT_NAME}
+                   DEPENDS ${CLIENT_LIB} ${SERVER_LIB} )
+
+
 #############################################################################
 
 # Things are a bit different on Macs when using an external libclang.dylib; here
 # we want to make sure we use @loader_path/libclang.dylib instead of
-# @rpath/libclang.dylib in the final ycm_core.so. If we use the @rpath version,
+# @rpath/libclang.dylib in the final ycm_core.so. If we use the
-# then it may load the system libclang which the user explicitely does not want
+# @rpath version, then it may load the system libclang which the user
-# (otherwise the user would specify USE_SYSTEM_LIBCLANG). With @loader_path, we
+# explicitely does not want (otherwise the user would specify
-# make sure that only the libclang.dylib present in the same directory as our
+# USE_SYSTEM_LIBCLANG). With @loader_path, we make sure that only the
-# ycm_core.so is used.
+# libclang.dylib present in the same directory as our ycm_core.so
+# is used.
 if ( EXTERNAL_LIBCLANG_PATH AND APPLE )
-  add_custom_command( TARGET ${PROJECT_NAME}
+  add_custom_command( TARGET ${SERVER_LIB}
                       POST_BUILD
                       COMMAND install_name_tool
                       "-change"
                       "@rpath/libclang.dylib"
                       "@loader_path/libclang.dylib"
-                      "$<TARGET_FILE:${PROJECT_NAME}>"
+                      "$<TARGET_FILE:${SERVER_LIB}>"
                     )
 endif()
 
@@ -268,19 +300,24 @@ endif()
 
 # We don't want the "lib" prefix, it can screw up python when it tries to search
 # for our module
-set_target_properties( ${PROJECT_NAME} PROPERTIES PREFIX "")
+set_target_properties( ${CLIENT_LIB} PROPERTIES PREFIX "")
+set_target_properties( ${SERVER_LIB} PROPERTIES PREFIX "")
 
 if ( WIN32 OR CYGWIN )
   # This is the extension for compiled Python modules on Windows
-  set_target_properties( ${PROJECT_NAME} PROPERTIES SUFFIX ".pyd")
+  set_target_properties( ${CLIENT_LIB} PROPERTIES SUFFIX ".pyd")
+  set_target_properties( ${SERVER_LIB} PROPERTIES SUFFIX ".pyd")
 else()
   # Even on macs, we want a .so extension instead of a .dylib which is what
   # cmake would give us by default. Python won't recognize a .dylib as a module,
   # but it will recognize a .so
-  set_target_properties( ${PROJECT_NAME} PROPERTIES SUFFIX ".so")
+  set_target_properties( ${CLIENT_LIB} PROPERTIES SUFFIX ".so")
+  set_target_properties( ${SERVER_LIB} PROPERTIES SUFFIX ".so")
 endif()
 
-set_target_properties( ${PROJECT_NAME} PROPERTIES
+set_target_properties( ${CLIENT_LIB} PROPERTIES
+  LIBRARY_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/../../python )
+set_target_properties( ${SERVER_LIB} PROPERTIES
   LIBRARY_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/../../python )
 
 #############################################################################

cpp/ycm/ClangCompleter/ClangCompleter.cpp

@@ -1,4 +1,4 @@
-// Copyright (C) 2011, 2012  Strahinja Val Markovic  <val@markovic.io>
+// Copyright (C) 2011-2013  Strahinja Val Markovic  <val@markovic.io>
 //
 // This file is part of YouCompleteMe.
 //
@@ -23,48 +23,22 @@
 #include "standard.h"
 #include "CandidateRepository.h"
 #include "CompletionData.h"
-#include "ConcurrentLatestValue.h"
 #include "Utils.h"
 #include "ClangUtils.h"
+#include "ReleaseGil.h"
 
 #include <clang-c/Index.h>
-#include <boost/make_shared.hpp>
+#include <boost/shared_ptr.hpp>
-#include <boost/algorithm/string.hpp>
-#include <boost/algorithm/cxx11/any_of.hpp>
 
-using boost::algorithm::any_of;
-using boost::algorithm::is_upper;
 
-using boost::bind;
-using boost::cref;
-using boost::function;
-using boost::lock_guard;
-using boost::make_shared;
-using boost::mutex;
-using boost::packaged_task;
-using boost::ref;
-using boost::shared_lock;
-using boost::shared_mutex;
 using boost::shared_ptr;
-using boost::thread;
-using boost::thread_interrupted;
-using boost::try_to_lock_t;
-using boost::unique_future;
-using boost::unique_lock;
 using boost::unordered_map;
 
 namespace YouCompleteMe {
 
-extern const unsigned int MAX_ASYNC_THREADS;
-extern const unsigned int MIN_ASYNC_THREADS;
-
 ClangCompleter::ClangCompleter()
   : clang_index_( clang_createIndex( 0, 0 ) ),
-    translation_unit_store_( clang_index_ ),
+    translation_unit_store_( clang_index_ ) {
-    candidate_repository_( CandidateRepository::Instance() ),
-    threading_enabled_( false ),
-    time_to_die_( false ),
-    clang_data_ready_( false ) {
   // The libclang docs don't say what is the default value for crash recovery.
   // I'm pretty sure it's turned on by default, but I'm not going to take any
   // chances.
@@ -73,47 +47,20 @@ ClangCompleter::ClangCompleter()
 
 
 ClangCompleter::~ClangCompleter() {
-  {
-    unique_lock< shared_mutex > lock( time_to_die_mutex_ );
-    time_to_die_ = true;
-  }
-
-  sorting_threads_.interrupt_all();
-  sorting_threads_.join_all();
-
-  if ( clang_thread_ ) {
-    clang_thread_->interrupt();
-    clang_thread_->join();
-  }
-
   // We need to destroy all TUs before calling clang_disposeIndex because
   // the translation units need to be destroyed before the index is destroyed.
+  // Technically, a thread could still be holding onto a shared_ptr<TU> object
+  // when we destroy the clang index, but since we're shutting down, we don't
+  // really care.
+  // In practice, this situation shouldn't happen because the server threads are
+  // Python deamon threads and will all be killed before the main thread exits.
   translation_unit_store_.RemoveAll();
   clang_disposeIndex( clang_index_ );
 }
 
 
-// We need this mostly so that we can not use it in tests. Apparently the
-// GoogleTest framework goes apeshit on us (on some platforms, in some
-// occasions) if we enable threads by default.
-void ClangCompleter::EnableThreading() {
-  threading_enabled_ = true;
-  InitThreads();
-}
-
-
-std::vector< Diagnostic > ClangCompleter::DiagnosticsForFile(
-  const std::string &filename ) {
-  shared_ptr< TranslationUnit > unit = translation_unit_store_.Get( filename );
-
-  if ( !unit )
-    return std::vector< Diagnostic >();
-
-  return unit->LatestDiagnostics();
-}
-
-
 bool ClangCompleter::UpdatingTranslationUnit( const std::string &filename ) {
+  ReleaseGil unlock;
   shared_ptr< TranslationUnit > unit = translation_unit_store_.Get( filename );
 
   if ( !unit )
@@ -126,10 +73,11 @@ bool ClangCompleter::UpdatingTranslationUnit( const std::string &filename ) {
 }
 
 
-void ClangCompleter::UpdateTranslationUnit(
+std::vector< Diagnostic > ClangCompleter::UpdateTranslationUnit(
   const std::string &filename,
   const std::vector< UnsavedFile > &unsaved_files,
   const std::vector< std::string > &flags ) {
+  ReleaseGil unlock;
   bool translation_unit_created;
   shared_ptr< TranslationUnit > unit = translation_unit_store_.GetOrCreate(
       filename,
@@ -138,13 +86,14 @@ void ClangCompleter::UpdateTranslationUnit(
       translation_unit_created );
 
   if ( !unit )
-    return;
+    return std::vector< Diagnostic >();
 
   try {
     // There's no point in reparsing a TU that was just created, it was just
     // parsed in the TU constructor
     if ( !translation_unit_created )
-      unit->Reparse( unsaved_files );
+      return unit->Reparse( unsaved_files );
+    return unit->LatestDiagnostics();
   }
 
   catch ( ClangParseError & ) {
@@ -153,30 +102,8 @@ void ClangCompleter::UpdateTranslationUnit(
     // TU map.
     translation_unit_store_.Remove( filename );
   }
-}
 
-
+  return std::vector< Diagnostic >();
-Future< void > ClangCompleter::UpdateTranslationUnitAsync(
-  std::string filename,
-  std::vector< UnsavedFile > unsaved_files,
-  std::vector< std::string > flags ) {
-  function< void() > functor =
-    boost::bind( &ClangCompleter::UpdateTranslationUnit,
-                 boost::ref( *this ),
-                 boost::move( filename ),
-                 boost::move( unsaved_files ),
-                 boost::move( flags ) );
-
-  shared_ptr< ClangPackagedTask > clang_packaged_task =
-    make_shared< ClangPackagedTask >();
-
-  clang_packaged_task->parsing_task_ = packaged_task< void >(
-                                         boost::move( functor ) );
-  unique_future< void > future =
-    clang_packaged_task->parsing_task_.get_future();
-  clang_task_.Set( clang_packaged_task );
-
-  return Future< void >( boost::move( future ) );
 }
 
 
@@ -187,6 +114,7 @@ ClangCompleter::CandidatesForLocationInFile(
   int column,
   const std::vector< UnsavedFile > &unsaved_files,
   const std::vector< std::string > &flags ) {
+  ReleaseGil unlock;
   shared_ptr< TranslationUnit > unit =
       translation_unit_store_.GetOrCreate( filename, unsaved_files, flags );
 
@@ -199,58 +127,13 @@ ClangCompleter::CandidatesForLocationInFile(
 }
 
 
-Future< AsyncCompletions >
-ClangCompleter::CandidatesForQueryAndLocationInFileAsync(
-  const std::string &query,
-  const std::string &filename,
-  int line,
-  int column,
-  const std::vector< UnsavedFile > &unsaved_files,
-  const std::vector< std::string > &flags ) {
-  // TODO: throw exception when threading is not enabled and this is called
-  if ( !threading_enabled_ )
-    return Future< AsyncCompletions >();
-
-  bool skip_clang_result_cache = ShouldSkipClangResultCache( query,
-                                                             line,
-                                                             column );
-
-  if ( skip_clang_result_cache ) {
-    // The clang thread is busy, return nothing
-    if ( UpdatingTranslationUnit( filename ) )
-      return Future< AsyncCompletions >();
-
-    {
-      lock_guard< mutex > lock( clang_data_ready_mutex_ );
-      clang_data_ready_ = false;
-    }
-
-    // Needed to "reset" the sorting threads to the start of their loop. This
-    // way any threads blocking on a read in sorting_task_.Get() are reset to
-    // wait on the clang_data_ready_condition_variable_.
-    sorting_threads_.interrupt_all();
-  }
-
-  // the sorting task needs to be set before the clang task (if any) just in
-  // case the clang task finishes (and therefore notifies a sorting thread to
-  // consume a sorting task) before the sorting task is set
-  unique_future< AsyncCompletions > future;
-  CreateSortingTask( query, future );
-
-  if ( skip_clang_result_cache ) {
-    CreateClangTask( filename, line, column, unsaved_files, flags );
-  }
-
-  return Future< AsyncCompletions >( boost::move( future ) );
-}
-
-
 Location ClangCompleter::GetDeclarationLocation(
   const std::string &filename,
   int line,
   int column,
   const std::vector< UnsavedFile > &unsaved_files,
   const std::vector< std::string > &flags ) {
+  ReleaseGil unlock;
   shared_ptr< TranslationUnit > unit =
       translation_unit_store_.GetOrCreate( filename, unsaved_files, flags );
 
@@ -268,6 +151,7 @@ Location ClangCompleter::GetDefinitionLocation(
   int column,
   const std::vector< UnsavedFile > &unsaved_files,
   const std::vector< std::string > &flags ) {
+  ReleaseGil unlock;
   shared_ptr< TranslationUnit > unit =
       translation_unit_store_.GetOrCreate( filename, unsaved_files, flags );
 
@@ -279,216 +163,9 @@ Location ClangCompleter::GetDefinitionLocation(
 }
 
 
-void ClangCompleter::DeleteCachesForFileAsync( const std::string &filename ) {
+void ClangCompleter::DeleteCachesForFile( const std::string &filename ) {
-  file_cache_delete_stack_.Push( filename );
+  ReleaseGil unlock;
-}
+  translation_unit_store_.Remove( filename );
-
-
-void ClangCompleter::DeleteCaches() {
-  std::vector< std::string > filenames;
-
-  if ( !file_cache_delete_stack_.PopAllNoWait( filenames ) )
-    return;
-
-  foreach( const std::string & filename, filenames ) {
-    translation_unit_store_.Remove( filename );
-  }
-}
-
-
-bool ClangCompleter::ShouldSkipClangResultCache( const std::string &query,
-                                                 int line,
-                                                 int column ) {
-  // We need query.empty() in addition to the second check because if we don't
-  // have it, then we have a problem in the following situation:
-  // The user has a variable 'foo' of type 'A' and a variable 'bar' of type 'B'.
-  // He then types in 'foo.' and we store the clang results and also return
-  // them. The user then deletes that code and types in 'bar.'. Without the
-  // query.empty() check we would return the results from the cache here (it's
-  // the same line and column!) which would be incorrect.
-  return query.empty() || latest_clang_results_
-         .NewPositionDifferentFromStoredPosition( line, column );
-}
-
-
-// Copy-ctor for unique_future is private in C++03 mode so we need to take it as
-// an out param
-void ClangCompleter::CreateSortingTask(
-  const std::string &query,
-  unique_future< AsyncCompletions > &future ) {
-  // Careful! The code in this function may burn your eyes.
-
-  function< CompletionDatas( const CompletionDatas & ) >
-  sort_candidates_for_query_functor =
-    boost::bind( &ClangCompleter::SortCandidatesForQuery,
-                 boost::ref( *this ),
-                 query,
-                 _1 );
-
-  function< CompletionDatas() > operate_on_completion_data_functor =
-    boost::bind( &ClangResultsCache::OperateOnCompletionDatas< CompletionDatas >,
-                 boost::cref( latest_clang_results_ ),
-                 boost::move( sort_candidates_for_query_functor ) );
-
-  shared_ptr< packaged_task< AsyncCompletions > > task =
-    boost::make_shared< packaged_task< AsyncCompletions > >(
-      boost::bind( ReturnValueAsShared< std::vector< CompletionData > >,
-                   boost::move( operate_on_completion_data_functor ) ) );
-
-  future = task->get_future();
-  sorting_task_.Set( task );
-}
-
-
-void ClangCompleter::CreateClangTask(
-  std::string filename,
-  int line,
-  int column,
-  std::vector< UnsavedFile > unsaved_files,
-  std::vector< std::string > flags ) {
-  latest_clang_results_.ResetWithNewLineAndColumn( line, column );
-
-  function< CompletionDatas() > candidates_for_location_functor =
-    boost::bind( &ClangCompleter::CandidatesForLocationInFile,
-                 boost::ref( *this ),
-                 boost::move( filename ),
-                 line,
-                 column,
-                 boost::move( unsaved_files ),
-                 boost::move( flags ) );
-
-  shared_ptr< ClangPackagedTask > clang_packaged_task =
-    make_shared< ClangPackagedTask >();
-
-  clang_packaged_task->completions_task_ =
-    packaged_task< AsyncCompletions >(
-      boost::bind( ReturnValueAsShared< std::vector< CompletionData > >,
-                   boost::move( candidates_for_location_functor ) ) );
-
-  clang_task_.Set( clang_packaged_task );
-}
-
-
-
-std::vector< CompletionData > ClangCompleter::SortCandidatesForQuery(
-  const std::string &query,
-  const std::vector< CompletionData > &completion_datas ) {
-  Bitset query_bitset = LetterBitsetFromString( query );
-  bool query_has_uppercase_letters = any_of( query, is_upper() );
-
-  std::vector< const Candidate * > repository_candidates =
-    candidate_repository_.GetCandidatesForStrings( completion_datas );
-
-  std::vector< ResultAnd< CompletionData * > > data_and_results;
-
-  for ( uint i = 0; i < repository_candidates.size(); ++i ) {
-    const Candidate *candidate = repository_candidates[ i ];
-
-    if ( !candidate->MatchesQueryBitset( query_bitset ) )
-      continue;
-
-    Result result = candidate->QueryMatchResult( query,
-                                                 query_has_uppercase_letters );
-
-    if ( result.IsSubsequence() ) {
-      ResultAnd< CompletionData * > data_and_result( &completion_datas[ i ],
-                                                     result );
-      data_and_results.push_back( boost::move( data_and_result ) );
-    }
-  }
-
-  std::sort( data_and_results.begin(), data_and_results.end() );
-
-  std::vector< CompletionData > sorted_completion_datas;
-  sorted_completion_datas.reserve( data_and_results.size() );
-
-  foreach ( const ResultAnd< CompletionData * > &data_and_result,
-            data_and_results ) {
-    sorted_completion_datas.push_back( *data_and_result.extra_object_ );
-  }
-
-  return sorted_completion_datas;
-}
-
-
-void ClangCompleter::InitThreads() {
-  int threads_to_create =
-    std::max( MIN_ASYNC_THREADS,
-              std::min( MAX_ASYNC_THREADS, thread::hardware_concurrency() ) );
-
-  for ( int i = 0; i < threads_to_create; ++i ) {
-    sorting_threads_.create_thread( bind( &ClangCompleter::SortingThreadMain,
-                                          boost::ref( *this ) ) );
-  }
-
-  clang_thread_.reset( new thread( &ClangCompleter::ClangThreadMain,
-                                   boost::ref( *this ) ) );
-}
-
-
-void ClangCompleter::ClangThreadMain() {
-  while ( true ) {
-    try {
-      shared_ptr< ClangPackagedTask > task = clang_task_.Get();
-
-      bool has_completions_task = task->completions_task_.valid();
-
-      if ( has_completions_task )
-        task->completions_task_();
-      else
-        task->parsing_task_();
-
-      if ( !has_completions_task ) {
-        DeleteCaches();
-        continue;
-      }
-
-      unique_future< AsyncCompletions > future =
-        task->completions_task_.get_future();
-      latest_clang_results_.SetCompletionDatas( *future.get() );
-
-      {
-        lock_guard< mutex > lock( clang_data_ready_mutex_ );
-        clang_data_ready_ = true;
-      }
-
-      clang_data_ready_condition_variable_.notify_all();
-    } catch ( thread_interrupted & ) {
-      shared_lock< shared_mutex > lock( time_to_die_mutex_ );
-
-      if ( time_to_die_ )
-        return;
-
-      // else do nothing and re-enter the loop
-    }
-  }
-}
-
-
-void ClangCompleter::SortingThreadMain() {
-  while ( true ) {
-    try {
-      {
-        unique_lock< mutex > lock( clang_data_ready_mutex_ );
-
-        while ( !clang_data_ready_ ) {
-          clang_data_ready_condition_variable_.wait( lock );
-        }
-      }
-
-      shared_ptr< packaged_task< AsyncCompletions > > task =
-        sorting_task_.Get();
-
-      ( *task )();
-    } catch ( thread_interrupted & ) {
-      shared_lock< shared_mutex > lock( time_to_die_mutex_ );
-
-      if ( time_to_die_ )
-        return;
-
-      // else do nothing and re-enter the loop
-    }
-  }
 }
 
 

cpp/ycm/ClangCompleter/ClangCompleter.h

@@ -18,18 +18,11 @@
 #ifndef CLANGCOMPLETE_H_WLKDU0ZV
 #define CLANGCOMPLETE_H_WLKDU0ZV
 
-#include "ConcurrentLatestValue.h"
-#include "ConcurrentStack.h"
-#include "Future.h"
 #include "UnsavedFile.h"
 #include "Diagnostic.h"
-#include "ClangResultsCache.h"
 #include "TranslationUnitStore.h"
 
 #include <boost/utility.hpp>
-#include <boost/thread/future.hpp>
-#include <boost/thread/mutex.hpp>
-#include <boost/unordered_map.hpp>
 
 #include <string>
 
@@ -37,46 +30,26 @@ typedef struct CXTranslationUnitImpl *CXTranslationUnit;
 
 namespace YouCompleteMe {
 
-class CandidateRepository;
 class TranslationUnit;
 struct CompletionData;
 struct Location;
 
 typedef std::vector< CompletionData > CompletionDatas;
 
-typedef boost::shared_ptr< CompletionDatas > AsyncCompletions;
 
-typedef boost::unordered_map < std::string,
+// All filename parameters must be absolute paths.
-        boost::shared_ptr <
-        std::vector< std::string > > > FlagsForFile;
-
-
-// TODO: document that all filename parameters must be absolute paths
 class ClangCompleter : boost::noncopyable {
 public:
   ClangCompleter();
   ~ClangCompleter();
 
-  void EnableThreading();
-
-  std::vector< Diagnostic > DiagnosticsForFile( const std::string &filename );
-
   bool UpdatingTranslationUnit( const std::string &filename );
 
-  // Public because of unit tests (gtest is not very thread-friendly)
+  std::vector< Diagnostic > UpdateTranslationUnit(
-  void UpdateTranslationUnit(
     const std::string &filename,
     const std::vector< UnsavedFile > &unsaved_files,
     const std::vector< std::string > &flags );
 
-  // NOTE: params are taken by value on purpose! With a C++11 compiler we can
-  // avoid internal copies if params are taken by value (move ctors FTW)
-  Future< void > UpdateTranslationUnitAsync(
-    std::string filename,
-    std::vector< UnsavedFile > unsaved_files,
-    std::vector< std::string > flags );
-
-  // Public because of unit tests (gtest is not very thread-friendly)
   std::vector< CompletionData > CandidatesForLocationInFile(
     const std::string &filename,
     int line,
@@ -84,14 +57,6 @@ public:
     const std::vector< UnsavedFile > &unsaved_files,
     const std::vector< std::string > &flags );
 
-  Future< AsyncCompletions > CandidatesForQueryAndLocationInFileAsync(
-    const std::string &query,
-    const std::string &filename,
-    int line,
-    int column,
-    const std::vector< UnsavedFile > &unsaved_files,
-    const std::vector< std::string > &flags );
-
   Location GetDeclarationLocation(
     const std::string &filename,
     int line,
@@ -106,55 +71,10 @@ public:
     const std::vector< UnsavedFile > &unsaved_files,
     const std::vector< std::string > &flags );
 
-  void DeleteCachesForFileAsync( const std::string &filename );
+  void DeleteCachesForFile( const std::string &filename );
 
 private:
 
-  void DeleteCaches();
-
-  // This is basically a union. Only one of the two tasks is set to something
-  // valid, the other task is invalid. Which one is valid depends on the caller.
-  struct ClangPackagedTask {
-    boost::packaged_task< AsyncCompletions > completions_task_;
-    boost::packaged_task< void > parsing_task_;
-  };
-
-  typedef ConcurrentLatestValue <
-  boost::shared_ptr <
-  boost::packaged_task< AsyncCompletions > > > LatestSortingTask;
-
-  typedef ConcurrentLatestValue <
-  boost::shared_ptr< ClangPackagedTask > > LatestClangTask;
-
-  typedef ConcurrentStack< std::string > FileCacheDeleteStack;
-
-  bool ShouldSkipClangResultCache( const std::string &query,
-                                   int line,
-                                   int column );
-
-  void CreateSortingTask( const std::string &query,
-                          boost::unique_future< AsyncCompletions > &future );
-
-  // NOTE: params are taken by value on purpose! With a C++11 compiler we can
-  // avoid internal copies if params are taken by value (move ctors FTW)
-  void CreateClangTask(
-    std::string filename,
-    int line,
-    int column,
-    std::vector< UnsavedFile > unsaved_files,
-    std::vector< std::string > flags );
-
-  std::vector< CompletionData > SortCandidatesForQuery(
-    const std::string &query,
-    const std::vector< CompletionData > &completion_datas );
-
-  void InitThreads();
-
-  void ClangThreadMain();
-
-  void SortingThreadMain();
-
-
   /////////////////////////////
   // PRIVATE MEMBER VARIABLES
   /////////////////////////////
@@ -162,35 +82,6 @@ private:
   CXIndex clang_index_;
 
   TranslationUnitStore translation_unit_store_;
-
-  CandidateRepository &candidate_repository_;
-
-  bool threading_enabled_;
-
-  // TODO: use boost.atomic for time_to_die_
-  bool time_to_die_;
-  boost::shared_mutex time_to_die_mutex_;
-
-  // TODO: use boost.atomic for clang_data_ready_
-  bool clang_data_ready_;
-  boost::mutex clang_data_ready_mutex_;
-  boost::condition_variable clang_data_ready_condition_variable_;
-
-  ClangResultsCache latest_clang_results_;
-
-  FileCacheDeleteStack file_cache_delete_stack_;
-
-  // Unfortunately clang is not thread-safe so we need to be careful when we
-  // access it. Only one thread at a time is allowed to access any single
-  // translation unit. Currently we only use one thread to access clang and that
-  // is the thread represented by clang_thread_.
-  boost::scoped_ptr< boost::thread > clang_thread_;
-
-  boost::thread_group sorting_threads_;
-
-  mutable LatestClangTask clang_task_;
-
-  mutable LatestSortingTask sorting_task_;
 };
 
 } // namespace YouCompleteMe

cpp/ycm/ClangCompleter/ClangResultsCache.h

@@ -1,84 +0,0 @@
-// Copyright (C) 2011, 2012  Strahinja Val Markovic  <val@markovic.io>
-//
-// This file is part of YouCompleteMe.
-//
-// YouCompleteMe is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// YouCompleteMe is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-//
-// You should have received a copy of the GNU General Public License
-// along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
-
-#ifndef CLANGRESULTSCACHE_H_REUWM3RU
-#define CLANGRESULTSCACHE_H_REUWM3RU
-
-#include "CompletionData.h"
-
-#include <vector>
-#include <boost/thread/locks.hpp>
-#include <boost/thread/shared_mutex.hpp>
-#include <boost/function.hpp>
-#include <boost/config.hpp>
-#include <boost/utility.hpp>
-
-namespace YouCompleteMe {
-
-struct CompletionData;
-
-class ClangResultsCache : boost::noncopyable {
-public:
-
-  ClangResultsCache() : line_( -1 ), column_( -1 ) {}
-
-  bool NewPositionDifferentFromStoredPosition( int new_line, int new_colum )
-  const;
-
-  void ResetWithNewLineAndColumn( int new_line, int new_colum );
-
-  void SetCompletionDatas(
-    const std::vector< CompletionData > new_completions ) {
-    completion_datas_ = new_completions;
-  }
-
-#ifndef BOOST_NO_RVALUE_REFERENCES
-#  ifdef __clang__
-#    pragma clang diagnostic push
-#    pragma clang diagnostic ignored "-Wc++98-compat"
-#  endif   //#ifdef __clang__
-
-  void SetCompletionDatas( std::vector< CompletionData >  &&new_completions ) {
-    completion_datas_ = new_completions;
-  }
-
-#  ifdef __clang__
-#    pragma clang diagnostic pop
-#  endif   //#ifdef __clang__
-#endif   //#ifndef BOOST_NO_RVALUE_REFERENCES
-
-  template< typename T >
-  T OperateOnCompletionDatas(
-    boost::function< T( const std::vector< CompletionData >& ) > operation )
-  const {
-    boost::shared_lock< boost::shared_mutex > reader_lock( access_mutex_ );
-    return operation( completion_datas_ );
-  }
-
-private:
-
-  int line_;
-  int column_;
-  std::vector< CompletionData > completion_datas_;
-
-  mutable boost::shared_mutex access_mutex_;
-};
-
-} // namespace YouCompleteMe
-
-#endif /* end of include guard: CLANGRESULTSCACHE_H_REUWM3RU */
-

cpp/ycm/ClangCompleter/ClangUtils.cpp

@@ -112,11 +112,8 @@ std::vector< CXUnsavedFile > ToCXUnsavedFiles(
   std::vector< CXUnsavedFile > clang_unsaved_files( unsaved_files.size() );
 
   for ( uint i = 0; i < unsaved_files.size(); ++i ) {
-    X_VERIFY( unsaved_files[ i ].filename_ );
+    clang_unsaved_files[ i ].Filename = unsaved_files[ i ].filename_.c_str();
-    X_VERIFY( unsaved_files[ i ].contents_ );
+    clang_unsaved_files[ i ].Contents = unsaved_files[ i ].contents_.c_str();
-    X_VERIFY( unsaved_files[ i ].length_ );
-    clang_unsaved_files[ i ].Filename = unsaved_files[ i ].filename_;
-    clang_unsaved_files[ i ].Contents = unsaved_files[ i ].contents_;
     clang_unsaved_files[ i ].Length   = unsaved_files[ i ].length_;
   }
 

cpp/ycm/ClangCompleter/ClangUtils.h

@@ -37,6 +37,8 @@ std::string CXStringToString( CXString text );
 std::vector< CompletionData > ToCompletionDataVector(
   CXCodeCompleteResults *results );
 
+// NOTE: CXUnsavedFiles store pointers to data in UnsavedFiles, so UnsavedFiles
+// need to outlive CXUnsavedFiles!
 std::vector< CXUnsavedFile > ToCXUnsavedFiles(
   const std::vector< UnsavedFile > &unsaved_files );
 

cpp/ycm/ClangCompleter/CompilationDatabase.cpp

@@ -18,20 +18,19 @@
 #include "CompilationDatabase.h"
 #include "ClangUtils.h"
 #include "standard.h"
+#include "ReleaseGil.h"
 
 #include <boost/shared_ptr.hpp>
-#include <boost/bind.hpp>
 #include <boost/make_shared.hpp>
 #include <boost/type_traits/remove_pointer.hpp>
+#include <boost/thread/locks.hpp>
 
-using boost::bind;
+using boost::lock_guard;
-using boost::make_shared;
+using boost::unique_lock;
-using boost::packaged_task;
+using boost::try_to_lock_t;
 using boost::remove_pointer;
 using boost::shared_ptr;
-using boost::thread;
+using boost::mutex;
-using boost::unique_future;
-using boost::function;
 
 namespace YouCompleteMe {
 
@@ -39,22 +38,9 @@ typedef shared_ptr <
 remove_pointer< CXCompileCommands >::type > CompileCommandsWrap;
 
 
-void QueryThreadMain( CompilationDatabase::InfoTaskStack &info_task_stack ) {
-  while ( true ) {
-    try {
-      ( *info_task_stack.Pop() )();
-    } catch ( boost::thread_interrupted & ) {
-      return;
-    }
-  }
-
-}
-
-
 CompilationDatabase::CompilationDatabase(
   const std::string &path_to_directory )
-  : threading_enabled_( false ),
+  : is_loaded_( false ) {
-    is_loaded_( false ) {
   CXCompilationDatabase_Error status;
   compilation_database_ = clang_CompilationDatabase_fromDirectory(
                             path_to_directory.c_str(),
@@ -68,27 +54,26 @@ CompilationDatabase::~CompilationDatabase() {
 }
 
 
-// We need this mostly so that we can not use it in tests. Apparently the
-// GoogleTest framework goes apeshit on us if we enable threads by default.
-void CompilationDatabase::EnableThreading() {
-  threading_enabled_ = true;
-  InitThreads();
-}
-
-
 bool CompilationDatabase::DatabaseSuccessfullyLoaded() {
   return is_loaded_;
 }
 
 
+bool CompilationDatabase::AlreadyGettingFlags() {
+  unique_lock< mutex > lock( compilation_database_mutex_, try_to_lock_t() );
+  return !lock.owns_lock();
+}
+
+
 CompilationInfoForFile CompilationDatabase::GetCompilationInfoForFile(
   const std::string &path_to_file ) {
+  ReleaseGil unlock;
   CompilationInfoForFile info;
 
   if ( !is_loaded_ )
     return info;
 
-  // TODO: mutex protect calls to getCompileCommands and getDirectory
+  lock_guard< mutex > lock( compilation_database_mutex_ );
 
   CompileCommandsWrap commands(
     clang_CompilationDatabase_getCompileCommands(
@@ -120,34 +105,5 @@ CompilationInfoForFile CompilationDatabase::GetCompilationInfoForFile(
   return info;
 }
 
-
-Future< AsyncCompilationInfoForFile >
-CompilationDatabase::GetCompilationInfoForFileAsync(
-  const std::string &path_to_file ) {
-  // TODO: throw exception when threading is not enabled and this is called
-  if ( !threading_enabled_ )
-    return Future< AsyncCompilationInfoForFile >();
-
-  function< CompilationInfoForFile() > functor =
-    boost::bind( &CompilationDatabase::GetCompilationInfoForFile,
-                 boost::ref( *this ),
-                 path_to_file );
-
-  InfoTask task =
-    make_shared< packaged_task< AsyncCompilationInfoForFile > >(
-      bind( ReturnValueAsShared< CompilationInfoForFile >,
-            functor ) );
-
-  unique_future< AsyncCompilationInfoForFile > future = task->get_future();
-  info_task_stack_.Push( task );
-  return Future< AsyncCompilationInfoForFile >( boost::move( future ) );
-}
-
-
-void CompilationDatabase::InitThreads() {
-  info_thread_ = boost::thread( QueryThreadMain,
-                                boost::ref( info_task_stack_ ) );
-}
-
 } // namespace YouCompleteMe
 

cpp/ycm/ClangCompleter/CompilationDatabase.h

@@ -18,15 +18,14 @@
 #ifndef COMPILATIONDATABASE_H_ZT7MQXPG
 #define COMPILATIONDATABASE_H_ZT7MQXPG
 
-#include "Future.h"
-#include "ConcurrentStack.h"
-
 #include <vector>
 #include <string>
 #include <boost/utility.hpp>
 #include <boost/shared_ptr.hpp>
+#include <boost/thread/mutex.hpp>
 #include <clang-c/CXCompilationDatabase.h>
 
+
 namespace YouCompleteMe {
 
 struct CompilationInfoForFile {
@@ -34,9 +33,8 @@ struct CompilationInfoForFile {
   std::string compiler_working_dir_;
 };
 
-typedef boost::shared_ptr< CompilationInfoForFile >
-AsyncCompilationInfoForFile;
 
+// Access to Clang's internal CompilationDatabase. This class is thread-safe.
 class CompilationDatabase : boost::noncopyable {
 public:
   CompilationDatabase( const std::string &path_to_directory );
@@ -44,28 +42,20 @@ public:
 
   bool DatabaseSuccessfullyLoaded();
 
-  void EnableThreading();
+  // Returns true when a separate thread is already getting flags; this is
+  // useful so that the caller doesn't need to block.
+  bool AlreadyGettingFlags();
 
+  // NOTE: Multiple calls to this function from separate threads will be
+  // serialized since Clang internals are not thread-safe.
   CompilationInfoForFile GetCompilationInfoForFile(
     const std::string &path_to_file );
 
-  Future< AsyncCompilationInfoForFile > GetCompilationInfoForFileAsync(
-    const std::string &path_to_file );
-
-  typedef boost::shared_ptr <
-  boost::packaged_task< AsyncCompilationInfoForFile > > InfoTask;
-
-  typedef ConcurrentStack< InfoTask > InfoTaskStack;
-
 private:
-  void InitThreads();
 
-  bool threading_enabled_;
   bool is_loaded_;
   CXCompilationDatabase compilation_database_;
-
+  boost::mutex compilation_database_mutex_;
-  boost::thread info_thread_;
-  InfoTaskStack info_task_stack_;
 };
 
 } // namespace YouCompleteMe

cpp/ycm/ClangCompleter/TranslationUnit.cpp

@@ -57,7 +57,7 @@ TranslationUnit::TranslationUnit(
   std::vector< CXUnsavedFile > cxunsaved_files =
     ToCXUnsavedFiles( unsaved_files );
   const CXUnsavedFile *unsaved = cxunsaved_files.size() > 0
-    ? &cxunsaved_files [0] : NULL;
+    ? &cxunsaved_files[ 0 ] : NULL;
 
   clang_translation_unit_ = clang_parseTranslationUnit(
                               clang_index,
@@ -92,25 +92,11 @@ void TranslationUnit::Destroy() {
 
 
 std::vector< Diagnostic > TranslationUnit::LatestDiagnostics() {
-  std::vector< Diagnostic > diagnostics;
-
   if ( !clang_translation_unit_ )
-    return diagnostics;
+    return std::vector< Diagnostic >();
 
   unique_lock< mutex > lock( diagnostics_mutex_ );
-
+  return latest_diagnostics_;
-  // We don't need the latest diags after we return them once so we swap the
-  // internal data with a new, empty diag vector. This vector is then returned
-  // and on C++11 compilers a move ctor is invoked, thus no copy is created.
-  // Theoretically, just returning the value of a
-  // [boost::|std::]move(latest_diagnostics_) call _should_ leave the
-  // latest_diagnostics_ vector in an emtpy, valid state but I'm not going to
-  // rely on that. I just had to look this up in the standard to be sure, and
-  // future readers of this code (myself included) should not be forced to do
-  // that to understand what the hell is going on.
-
-  std::swap( latest_diagnostics_, diagnostics );
-  return diagnostics;
 }
 
 
@@ -125,12 +111,15 @@ bool TranslationUnit::IsCurrentlyUpdating() const {
 }
 
 
-void TranslationUnit::Reparse(
+std::vector< Diagnostic > TranslationUnit::Reparse(
   const std::vector< UnsavedFile > &unsaved_files ) {
   std::vector< CXUnsavedFile > cxunsaved_files =
     ToCXUnsavedFiles( unsaved_files );
 
   Reparse( cxunsaved_files );
+
+  unique_lock< mutex > lock( diagnostics_mutex_ );
+  return latest_diagnostics_;
 }
 
 
@@ -157,7 +146,8 @@ std::vector< CompletionData > TranslationUnit::CandidatesForLocation(
   std::vector< CXUnsavedFile > cxunsaved_files =
     ToCXUnsavedFiles( unsaved_files );
   const CXUnsavedFile *unsaved = cxunsaved_files.size() > 0
-    ? &cxunsaved_files [0] : NULL;
+    ? &cxunsaved_files[ 0 ] : NULL;
+
   // codeCompleteAt reparses the TU if the underlying source file has changed on
   // disk since the last time the TU was updated and there are no unsaved files.
   // If there are unsaved files, then codeCompleteAt will parse the in-memory
@@ -253,7 +243,8 @@ void TranslationUnit::Reparse( std::vector< CXUnsavedFile > &unsaved_files,
     if ( !clang_translation_unit_ )
       return;
     CXUnsavedFile *unsaved = unsaved_files.size() > 0
-      ? &unsaved_files [0] : NULL;
+      ? &unsaved_files[ 0 ] : NULL;
+
     failure = clang_reparseTranslationUnit( clang_translation_unit_,
                                             unsaved_files.size(),
                                             unsaved,

cpp/ycm/ClangCompleter/TranslationUnit.h

@@ -18,8 +18,6 @@
 #ifndef TRANSLATIONUNIT_H_XQ7I6SVA
 #define TRANSLATIONUNIT_H_XQ7I6SVA
 
-#include "ConcurrentLatestValue.h"
-#include "Future.h"
 #include "UnsavedFile.h"
 #include "Diagnostic.h"
 #include "Location.h"
@@ -58,7 +56,8 @@ public:
 
   bool IsCurrentlyUpdating() const;
 
-  void Reparse( const std::vector< UnsavedFile > &unsaved_files );
+  std::vector< Diagnostic > Reparse(
+      const std::vector< UnsavedFile > &unsaved_files );
 
   void ReparseForIndexing( const std::vector< UnsavedFile > &unsaved_files );
 

cpp/ycm/ClangCompleter/TranslationUnitStore.cpp

@@ -21,6 +21,7 @@
 #include "exceptions.h"
 
 #include <boost/thread/locks.hpp>
+#include <boost/make_shared.hpp>
 #include <boost/functional/hash.hpp>
 
 using boost::lock_guard;
@@ -38,14 +39,17 @@ std::size_t HashForFlags( const std::vector< std::string > &flags ) {
 
 }  // unnamed namespace
 
+
 TranslationUnitStore::TranslationUnitStore( CXIndex clang_index )
   : clang_index_( clang_index ) {
 }
 
+
 TranslationUnitStore::~TranslationUnitStore() {
   RemoveAll();
 }
 
+
 shared_ptr< TranslationUnit > TranslationUnitStore::GetOrCreate(
   const std::string &filename,
   const std::vector< UnsavedFile > &unsaved_files,
@@ -104,24 +108,28 @@ shared_ptr< TranslationUnit > TranslationUnitStore::GetOrCreate(
   return unit;
 }
 
+
 shared_ptr< TranslationUnit > TranslationUnitStore::Get(
     const std::string &filename ) {
   lock_guard< mutex > lock( filename_to_translation_unit_and_flags_mutex_ );
   return GetNoLock( filename );
 }
 
+
 bool TranslationUnitStore::Remove( const std::string &filename ) {
   lock_guard< mutex > lock( filename_to_translation_unit_and_flags_mutex_ );
   Erase( filename_to_flags_hash_, filename );
   return Erase( filename_to_translation_unit_, filename );
 }
 
+
 void TranslationUnitStore::RemoveAll() {
   lock_guard< mutex > lock( filename_to_translation_unit_and_flags_mutex_ );
   filename_to_translation_unit_.clear();
   filename_to_flags_hash_.clear();
 }
 
+
 shared_ptr< TranslationUnit > TranslationUnitStore::GetNoLock(
     const std::string &filename ) {
   return FindWithDefault( filename_to_translation_unit_,

cpp/ycm/ClangCompleter/UnsavedFile.h

@@ -18,13 +18,13 @@
 #ifndef UNSAVEDFILE_H_0GIYZQL4
 #define UNSAVEDFILE_H_0GIYZQL4
 
-#include <cstddef>
+#include <string>
 
 struct UnsavedFile {
-  UnsavedFile() : filename_( NULL ), contents_( NULL ), length_( 0 ) {}
+  UnsavedFile() : filename_( "" ), contents_( "" ), length_( 0 ) {}
 
-  const char *filename_;
+  std::string filename_;
-  const char *contents_;
+  std::string contents_;
   unsigned long length_;
 
   // We need this to be able to export this struct to Python via Boost.Python's

cpp/ycm/ConcurrentLatestValue.h

@@ -1,79 +0,0 @@
-// Copyright (C) 2011, 2012  Strahinja Val Markovic  <val@markovic.io>
-//
-// This file is part of YouCompleteMe.
-//
-// YouCompleteMe is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// YouCompleteMe is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-//
-// You should have received a copy of the GNU General Public License
-// along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
-
-#ifndef CONCURRENTLATESTVALUE_H_SYF1JPPG
-#define CONCURRENTLATESTVALUE_H_SYF1JPPG
-
-#include <boost/thread.hpp>
-#include <boost/utility.hpp>
-
-namespace YouCompleteMe {
-
-// This is is basically a multi-consumer single-producer queue, only with the
-// twist that we only care about the latest value set. So the GUI thread is the
-// setter, and the worker threads are the workers. The workers wait in line on
-// the condition variable and when the setter sets a value, a worker is chosen
-// to consume it.
-//
-// The point is that we always want to have one "fresh" worker thread ready to
-// work on our latest value. If a newer value is set, then we don't care what
-// happens to the old values.
-//
-// This implementation is mutex-based and is not lock-free. Normally using a
-// lock-free data structure makes more sense, but since the GUI thread goes
-// through VimL and Python on every keystroke, there's really no point. Those 5
-// nanoseconds it takes to lock a mutex are laughably negligible compared to the
-// VimL/Python overhead.
-template <typename T>
-class ConcurrentLatestValue : boost::noncopyable {
-public:
-
-  ConcurrentLatestValue() : empty_( true ) {}
-
-  void Set( const T &data ) {
-    {
-      boost::unique_lock< boost::mutex > lock( mutex_ );
-      latest_ = data;
-      empty_ = false;
-    }
-
-    condition_variable_.notify_one();
-  }
-
-  T Get() {
-    boost::unique_lock< boost::mutex > lock( mutex_ );
-
-    while ( empty_ ) {
-      condition_variable_.wait( lock );
-    }
-
-    empty_ = true;
-    return latest_;
-  }
-
-
-private:
-  T latest_;
-  bool empty_;
-  boost::mutex mutex_;
-  boost::condition_variable condition_variable_;
-
-};
-
-} // namespace YouCompleteMe
-
-#endif /* end of include guard: CONCURRENTLATESTVALUE_H_SYF1JPPG */

cpp/ycm/ConcurrentStack.h

@@ -1,89 +0,0 @@
-// Copyright (C) 2011, 2012  Strahinja Val Markovic  <val@markovic.io>
-//
-// This file is part of YouCompleteMe.
-//
-// YouCompleteMe is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// YouCompleteMe is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-//
-// You should have received a copy of the GNU General Public License
-// along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
-
-#ifndef CONCURRENTSTACK_H_TGI0GOR6
-#define CONCURRENTSTACK_H_TGI0GOR6
-
-#include <boost/thread.hpp>
-#include <boost/utility.hpp>
-#include <stack>
-
-namespace YouCompleteMe {
-
-template <typename T>
-class ConcurrentStack : boost::noncopyable {
-public:
-
-  void Push( const T &data ) {
-    {
-      boost::unique_lock< boost::mutex > lock( mutex_ );
-      stack_.push( data );
-    }
-
-    condition_variable_.notify_one();
-  }
-
-
-  T Pop() {
-    boost::unique_lock< boost::mutex > lock( mutex_ );
-
-    while ( stack_.empty() ) {
-      condition_variable_.wait( lock );
-    }
-
-    T top = stack_.top();
-    stack_.pop();
-    return top;
-  }
-
-
-  // Gets all the items from the stack and appends them to the input vector.
-  // Does not wait for the stack to get items; if the stack is empty, returns
-  // false and does not touch the input vector.
-  bool PopAllNoWait( std::vector< T > &items ) {
-    boost::unique_lock< boost::mutex > lock( mutex_ );
-
-    if ( stack_.empty() )
-      return false;
-
-    int num_items = stack_.size();
-    items.reserve( num_items + items.size() );
-
-    for ( int i = 0; i < num_items; ++i ) {
-      items.push_back( stack_.top() );
-      stack_.pop();
-    }
-
-    return true;
-  }
-
-
-  bool Empty() {
-    boost::unique_lock< boost::mutex > lock( mutex_ );
-    return stack_.empty();
-  }
-
-private:
-  std::stack< T > stack_;
-  boost::mutex mutex_;
-  boost::condition_variable condition_variable_;
-
-};
-
-} // namespace YouCompleteMe
-
-#endif /* end of include guard: CONCURRENTSTACK_H_TGI0GOR6 */

cpp/ycm/Future.h

@@ -1,78 +0,0 @@
-// Copyright (C) 2011, 2012  Strahinja Val Markovic  <val@markovic.io>
-//
-// This file is part of YouCompleteMe.
-//
-// YouCompleteMe is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// YouCompleteMe is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-//
-// You should have received a copy of the GNU General Public License
-// along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
-
-#ifndef FUTURE_H_NR1U6MZS
-#define FUTURE_H_NR1U6MZS
-
-#include <boost/thread.hpp>
-#include <boost/make_shared.hpp>
-#include <boost/shared_ptr.hpp>
-#include <boost/function.hpp>
-
-namespace YouCompleteMe {
-
-class Result;
-typedef boost::shared_ptr< boost::packaged_task< void > > VoidTask;
-
-template< typename T >
-boost::shared_ptr< T > ReturnValueAsShared(
-  boost::function< T() > func ) {
-  return boost::make_shared< T >( func() );
-}
-
-
-template< typename T >
-class Future {
-public:
-  Future() {};
-  Future( boost::shared_future< T > future )
-    : future_( boost::move( future ) ) {}
-
-  bool ResultsReady() {
-    // It's OK to return true since GetResults will just return a
-    // default-constructed value if the future_ is uninitialized. If we don't
-    // return true for this case, any loop waiting on ResultsReady will wait
-    // forever.
-    if ( future_.get_state() == boost::future_state::uninitialized )
-      return true;
-
-    return future_.is_ready();
-  }
-
-
-  void Wait() {
-    future_.wait();
-  }
-
-
-  T GetResults() {
-    try {
-      return future_.get();
-    } catch ( boost::future_uninitialized & ) {
-      // Do nothing and return a T()
-    }
-
-    return T();
-  }
-
-private:
-  boost::shared_future< T > future_;
-};
-
-} // namespace YouCompleteMe
-
-#endif /* end of include guard: FUTURE_H_NR1U6MZS */

cpp/ycm/IdentifierCompleter.cpp

@@ -22,63 +22,18 @@
 #include "IdentifierUtils.h"
 #include "Result.h"
 #include "Utils.h"
+#include "ReleaseGil.h"
 
-#include <boost/bind.hpp>
-#include <boost/make_shared.hpp>
 #include <algorithm>
 
-using boost::packaged_task;
-using boost::unique_future;
-using boost::shared_ptr;
-using boost::thread;
-
 namespace YouCompleteMe {
 
-typedef boost::function< std::vector< std::string >() >
-FunctionReturnsStringVector;
 
-
+IdentifierCompleter::IdentifierCompleter() {}
-extern const unsigned int MAX_ASYNC_THREADS = 4;
-extern const unsigned int MIN_ASYNC_THREADS = 2;
-
-namespace {
-
-void QueryThreadMain(
-  IdentifierCompleter::LatestQueryTask &latest_query_task ) {
-  while ( true ) {
-    try {
-      ( *latest_query_task.Get() )();
-    } catch ( boost::thread_interrupted & ) {
-      return;
-    }
-  }
-
-}
-
-void BufferIdentifiersThreadMain(
-  IdentifierCompleter::BufferIdentifiersTaskStack
-  &buffer_identifiers_task_stack ) {
-  while ( true ) {
-    try {
-      ( *buffer_identifiers_task_stack.Pop() )();
-    } catch ( boost::thread_interrupted & ) {
-      return;
-    }
-  }
-}
-
-
-} // unnamed namespace
-
-
-IdentifierCompleter::IdentifierCompleter()
-  : threading_enabled_( false ) {
-}
 
 
 IdentifierCompleter::IdentifierCompleter(
-  const std::vector< std::string > &candidates )
+  const std::vector< std::string > &candidates ) {
-  : threading_enabled_( false ) {
   identifier_database_.AddIdentifiers( candidates, "", "" );
 }
 
@@ -86,35 +41,16 @@ IdentifierCompleter::IdentifierCompleter(
 IdentifierCompleter::IdentifierCompleter(
   const std::vector< std::string > &candidates,
   const std::string &filetype,
-  const std::string &filepath )
+  const std::string &filepath ) {
-  : threading_enabled_( false ) {
   identifier_database_.AddIdentifiers( candidates, filetype, filepath );
 }
 
 
-IdentifierCompleter::~IdentifierCompleter() {
-  query_threads_.interrupt_all();
-  query_threads_.join_all();
-
-  if ( buffer_identifiers_thread_ ) {
-    buffer_identifiers_thread_->interrupt();
-    buffer_identifiers_thread_->join();
-  }
-}
-
-
-// We need this mostly so that we can not use it in tests. Apparently the
-// GoogleTest framework goes apeshit on us if we enable threads by default.
-void IdentifierCompleter::EnableThreading() {
-  threading_enabled_ = true;
-  InitThreads();
-}
-
-
 void IdentifierCompleter::AddIdentifiersToDatabase(
   const std::vector< std::string > &new_candidates,
   const std::string &filetype,
   const std::string &filepath ) {
+  ReleaseGil unlock;
   identifier_database_.AddIdentifiers( new_candidates,
                                        filetype,
                                        filepath );
@@ -123,6 +59,7 @@ void IdentifierCompleter::AddIdentifiersToDatabase(
 
 void IdentifierCompleter::AddIdentifiersToDatabaseFromTagFiles(
   const std::vector< std::string > &absolute_paths_to_tag_files ) {
+  ReleaseGil unlock;
   foreach( const std::string & path, absolute_paths_to_tag_files ) {
     identifier_database_.AddIdentifiers(
       ExtractIdentifiersFromTagsFile( path ) );
@@ -130,27 +67,12 @@ void IdentifierCompleter::AddIdentifiersToDatabaseFromTagFiles(
 }
 
 
-void IdentifierCompleter::AddIdentifiersToDatabaseFromTagFilesAsync(
-  std::vector< std::string > absolute_paths_to_tag_files ) {
-  // TODO: throw exception when threading is not enabled and this is called
-  if ( !threading_enabled_ )
-    return;
-
-  boost::function< void() > functor =
-    boost::bind( &IdentifierCompleter::AddIdentifiersToDatabaseFromTagFiles,
-                 boost::ref( *this ),
-                 boost::move( absolute_paths_to_tag_files ) );
-
-  buffer_identifiers_task_stack_.Push(
-    boost::make_shared< packaged_task< void > >( boost::move( functor ) ) );
-}
-
-
 void IdentifierCompleter::AddIdentifiersToDatabaseFromBuffer(
   const std::string &buffer_contents,
   const std::string &filetype,
   const std::string &filepath,
   bool collect_from_comments_and_strings ) {
+  ReleaseGil unlock;
   identifier_database_.ClearCandidatesStoredForFile( filetype, filepath );
 
   std::string new_contents =
@@ -165,28 +87,6 @@ void IdentifierCompleter::AddIdentifiersToDatabaseFromBuffer(
 }
 
 
-void IdentifierCompleter::AddIdentifiersToDatabaseFromBufferAsync(
-  std::string buffer_contents,
-  std::string filetype,
-  std::string filepath,
-  bool collect_from_comments_and_strings ) {
-  // TODO: throw exception when threading is not enabled and this is called
-  if ( !threading_enabled_ )
-    return;
-
-  boost::function< void() > functor =
-    boost::bind( &IdentifierCompleter::AddIdentifiersToDatabaseFromBuffer,
-                 boost::ref( *this ),
-                 boost::move( buffer_contents ),
-                 boost::move( filetype ),
-                 boost::move( filepath ),
-                 collect_from_comments_and_strings );
-
-  buffer_identifiers_task_stack_.Push(
-    boost::make_shared< packaged_task< void > >( boost::move( functor ) ) );
-}
-
-
 std::vector< std::string > IdentifierCompleter::CandidatesForQuery(
   const std::string &query ) const {
   return CandidatesForQueryAndType( query, "" );
@@ -196,6 +96,7 @@ std::vector< std::string > IdentifierCompleter::CandidatesForQuery(
 std::vector< std::string > IdentifierCompleter::CandidatesForQueryAndType(
   const std::string &query,
   const std::string &filetype ) const {
+  ReleaseGil unlock;
   std::vector< Result > results;
   identifier_database_.ResultsForQueryAndType( query, filetype, results );
 
@@ -209,46 +110,4 @@ std::vector< std::string > IdentifierCompleter::CandidatesForQueryAndType(
 }
 
 
-Future< AsyncResults > IdentifierCompleter::CandidatesForQueryAndTypeAsync(
-  const std::string &query,
-  const std::string &filetype ) const {
-  // TODO: throw exception when threading is not enabled and this is called
-  if ( !threading_enabled_ )
-    return Future< AsyncResults >();
-
-  FunctionReturnsStringVector functor =
-    boost::bind( &IdentifierCompleter::CandidatesForQueryAndType,
-                 boost::cref( *this ),
-                 query,
-                 filetype );
-
-  QueryTask task =
-    boost::make_shared< packaged_task< AsyncResults > >(
-      boost::bind( ReturnValueAsShared< std::vector< std::string > >,
-                   boost::move( functor ) ) );
-
-  unique_future< AsyncResults > future = task->get_future();
-
-  latest_query_task_.Set( task );
-  return Future< AsyncResults >( boost::move( future ) );
-}
-
-
-void IdentifierCompleter::InitThreads() {
-  int query_threads_to_create =
-    std::max( MIN_ASYNC_THREADS,
-              std::min( MAX_ASYNC_THREADS, thread::hardware_concurrency() ) );
-
-  for ( int i = 0; i < query_threads_to_create; ++i ) {
-    query_threads_.create_thread(
-      boost::bind( QueryThreadMain,
-                   boost::ref( latest_query_task_ ) ) );
-  }
-
-  buffer_identifiers_thread_.reset(
-    new boost::thread( BufferIdentifiersThreadMain,
-                       boost::ref( buffer_identifiers_task_stack_ ) ) );
-}
-
-
 } // namespace YouCompleteMe

cpp/ycm/IdentifierCompleter.h

@@ -18,10 +18,7 @@
 #ifndef COMPLETER_H_7AR4UGXE
 #define COMPLETER_H_7AR4UGXE
 
-#include "ConcurrentLatestValue.h"
-#include "ConcurrentStack.h"
 #include "IdentifierDatabase.h"
-#include "Future.h"
 
 #include <boost/utility.hpp>
 #include <boost/unordered_map.hpp>
@@ -36,8 +33,6 @@ namespace YouCompleteMe {
 
 class Candidate;
 
-typedef boost::shared_ptr< std::vector< std::string > > AsyncResults;
-
 
 class IdentifierCompleter : boost::noncopyable {
 public:
@@ -47,10 +42,6 @@ public:
                        const std::string &filetype,
                        const std::string &filepath );
 
-  ~IdentifierCompleter();
-
-  void EnableThreading();
-
   void AddIdentifiersToDatabase(
     const std::vector< std::string > &new_candidates,
     const std::string &filetype,
@@ -59,25 +50,12 @@ public:
   void AddIdentifiersToDatabaseFromTagFiles(
     const std::vector< std::string > &absolute_paths_to_tag_files );
 
-  // NOTE: params are taken by value on purpose!
-  void AddIdentifiersToDatabaseFromTagFilesAsync(
-    std::vector< std::string > absolute_paths_to_tag_files );
-
   void AddIdentifiersToDatabaseFromBuffer(
     const std::string &buffer_contents,
     const std::string &filetype,
     const std::string &filepath,
     bool collect_from_comments_and_strings );
 
-  // NOTE: params are taken by value on purpose! With a C++11 compiler we can
-  // avoid an expensive copy of buffer_contents if the param is taken by value
-  // (move ctors FTW)
-  void AddIdentifiersToDatabaseFromBufferAsync(
-    std::string buffer_contents,
-    std::string filetype,
-    std::string filepath,
-    bool collect_from_comments_and_strings );
-
   // Only provided for tests!
   std::vector< std::string > CandidatesForQuery(
     const std::string &query ) const;
@@ -86,37 +64,13 @@ public:
     const std::string &query,
     const std::string &filetype ) const;
 
-  Future< AsyncResults > CandidatesForQueryAndTypeAsync(
-    const std::string &query,
-    const std::string &filetype ) const;
-
-  typedef boost::shared_ptr <
-  boost::packaged_task< AsyncResults > > QueryTask;
-
-  typedef ConcurrentLatestValue< QueryTask > LatestQueryTask;
-
-  typedef ConcurrentStack< VoidTask > BufferIdentifiersTaskStack;
-
 private:
 
-  void InitThreads();
-
-
   /////////////////////////////
   // PRIVATE MEMBER VARIABLES
   /////////////////////////////
 
   IdentifierDatabase identifier_database_;
-
-  bool threading_enabled_;
-
-  boost::thread_group query_threads_;
-
-  boost::scoped_ptr< boost::thread > buffer_identifiers_thread_;
-
-  mutable LatestQueryTask latest_query_task_;
-
-  BufferIdentifiersTaskStack buffer_identifiers_task_stack_;
 };
 
 } // namespace YouCompleteMe

cpp/ycm/PythonSupport.cpp

@@ -20,6 +20,8 @@
 #include "Result.h"
 #include "Candidate.h"
 #include "CandidateRepository.h"
+#include "ReleaseGil.h"
+
 #include <boost/algorithm/string.hpp>
 #include <boost/algorithm/cxx11/any_of.hpp>
 #include <vector>
@@ -27,6 +29,7 @@
 using boost::algorithm::any_of;
 using boost::algorithm::is_upper;
 using boost::python::len;
+using boost::python::str;
 using boost::python::extract;
 using boost::python::object;
 typedef boost::python::list pylist;
@@ -35,6 +38,13 @@ namespace YouCompleteMe {
 
 namespace {
 
+std::string GetUtf8String( const boost::python::object &string_or_unicode ) {
+  extract< std::string > to_string( string_or_unicode );
+  if ( to_string.check() )
+    return to_string();
+  return extract< std::string >( str( string_or_unicode ).encode( "utf8" ) );
+}
+
 std::vector< const Candidate * > CandidatesFromObjectList(
   const pylist &candidates,
   const std::string &candidate_property ) {
@@ -44,10 +54,10 @@ std::vector< const Candidate * > CandidatesFromObjectList(
 
   for ( int i = 0; i < num_candidates; ++i ) {
     if ( candidate_property.empty() ) {
-      candidate_strings.push_back( extract< std::string >( candidates[ i ] ) );
+      candidate_strings.push_back( GetUtf8String( candidates[ i ] ) );
     } else {
       object holder = extract< object >( candidates[ i ] );
-      candidate_strings.push_back( extract< std::string >(
+      candidate_strings.push_back( GetUtf8String(
                                      holder[ candidate_property.c_str() ] ) );
     }
   }
@@ -68,31 +78,33 @@ boost::python::list FilterAndSortCandidates(
     return candidates;
   }
 
+  int num_candidates = len( candidates );
   std::vector< const Candidate * > repository_candidates =
     CandidatesFromObjectList( candidates, candidate_property );
 
-  Bitset query_bitset = LetterBitsetFromString( query );
-  bool query_has_uppercase_letters = any_of( query, is_upper() );
-
-  int num_candidates = len( candidates );
   std::vector< ResultAnd< int > > object_and_results;
+  {
+    ReleaseGil unlock;
+    Bitset query_bitset = LetterBitsetFromString( query );
+    bool query_has_uppercase_letters = any_of( query, is_upper() );
 
-  for ( int i = 0; i < num_candidates; ++i ) {
+    for ( int i = 0; i < num_candidates; ++i ) {
-    const Candidate *candidate = repository_candidates[ i ];
+      const Candidate *candidate = repository_candidates[ i ];
 
-    if ( !candidate->MatchesQueryBitset( query_bitset ) )
+      if ( !candidate->MatchesQueryBitset( query_bitset ) )
-      continue;
+        continue;
 
-    Result result = candidate->QueryMatchResult( query,
+      Result result = candidate->QueryMatchResult( query,
-                                                 query_has_uppercase_letters );
+                                                  query_has_uppercase_letters );
 
-    if ( result.IsSubsequence() ) {
+      if ( result.IsSubsequence() ) {
-      ResultAnd< int > object_and_result( i, result );
+        ResultAnd< int > object_and_result( i, result );
-      object_and_results.push_back( boost::move( object_and_result ) );
+        object_and_results.push_back( boost::move( object_and_result ) );
+      }
     }
-  }
 
-  std::sort( object_and_results.begin(), object_and_results.end() );
+    std::sort( object_and_results.begin(), object_and_results.end() );
+  }
 
   foreach ( const ResultAnd< int > &object_and_result,
             object_and_results ) {

cpp/ycm/ReleaseGil.h

@@ -1,4 +1,4 @@
-// Copyright (C) 2011, 2012  Strahinja Val Markovic  <val@markovic.io>
+// Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
 //
 // This file is part of YouCompleteMe.
 //
@@ -15,29 +15,28 @@
 // You should have received a copy of the GNU General Public License
 // along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
-#include "ClangResultsCache.h"
+#ifndef RELEASEGIL_H_RDIEBSQ1
-#include "standard.h"
+#define RELEASEGIL_H_RDIEBSQ1
 
-using boost::shared_mutex;
+#include <boost/python.hpp>
-using boost::shared_lock;
-using boost::unique_lock;
 
 namespace YouCompleteMe {
 
-bool ClangResultsCache::NewPositionDifferentFromStoredPosition( int new_line,
+class ReleaseGil {
-                                                                int new_colum )
+public:
-const {
+  ReleaseGil() {
-  shared_lock< shared_mutex > reader_lock( access_mutex_ );
+    thread_state_ = PyEval_SaveThread();
-  return line_ != new_line || column_ != new_colum;
+  }
-}
 
-void ClangResultsCache::ResetWithNewLineAndColumn( int new_line,
+  ~ReleaseGil() {
-                                                   int new_colum ) {
+    PyEval_RestoreThread( thread_state_ );
-  unique_lock< shared_mutex > reader_lock( access_mutex_ );
+  }
 
-  line_ = new_line;
+private:
-  column_ = new_colum;
+  PyThreadState *thread_state_;
-  completion_datas_.clear();
+};
-}
 
 } // namespace YouCompleteMe
+
+#endif /* end of include guard: RELEASEGIL_H_RDIEBSQ1 */
+

cpp/ycm/Result.h

@@ -88,6 +88,7 @@ private:
 
 template< class T >
 struct ResultAnd {
+  // TODO: Swap the order of these parameters
   ResultAnd( T extra_object, const Result &result )
     : extra_object_( extra_object ), result_( result ) {}
 

cpp/ycm/tests/CMakeLists.txt

@@ -28,7 +28,7 @@ endif()
 add_subdirectory( gmock )
 
 include_directories(
-  ${ycm_core_SOURCE_DIR}
+  ${ycm_support_libs_SOURCE_DIR}
   )
 
 include_directories(
@@ -67,8 +67,9 @@ add_executable( ${PROJECT_NAME}
               )
 
 target_link_libraries( ${PROJECT_NAME}
-                       ycm_core
+                       ${SERVER_LIB}
-                       gmock_main )
+                       ${CLIENT_LIB}
+                       gmock )
 
 
 if ( NOT CMAKE_GENERATOR_IS_XCODE )

cpp/ycm/tests/ClangCompleter/ClangCompleter_test.cpp

@@ -43,22 +43,21 @@ TEST( ClangCompleterTest, CandidatesForLocationInFile ) {
 }
 
 
-TEST( ClangCompleterTest, CandidatesForQueryAndLocationInFileAsync ) {
+TEST( ClangCompleterTest, GetDefinitionLocation ) {
   ClangCompleter completer;
-  completer.EnableThreading();
+  std::string filename = PathToTestFile( "basic.cpp" ).string();
 
-  Future< AsyncCompletions > completions_future =
+  // Clang operates on the reasonable assumption that line and column numbers
-    completer.CandidatesForQueryAndLocationInFileAsync(
+  // are 1-based.
-      "",
+  Location actual_location =
-      PathToTestFile( "basic.cpp" ).string(),
+    completer.GetDefinitionLocation(
-      11,
+      filename,
-      7,
+      9,
+      3,
       std::vector< UnsavedFile >(),
       std::vector< std::string >() );
 
-  completions_future.Wait();
+  EXPECT_EQ( Location( filename, 1, 8 ), actual_location );
-
-  EXPECT_TRUE( !completions_future.GetResults()->empty() );
 }
 
 } // namespace YouCompleteMe

cpp/ycm/tests/IdentifierCompleter_test.cpp

@@ -217,6 +217,18 @@ TEST( IdentifierCompleterTest, ShorterAndLowercaseWins ) {
                             "STDIN_FILENO" ) );
 }
 
+TEST( IdentifierCompleterTest, AddIdentifiersToDatabaseFromBufferWorks ) {
+  IdentifierCompleter completer;
+  completer.AddIdentifiersToDatabaseFromBuffer( "foo foogoo ba",
+                                                "foo",
+                                                "/foo/bar",
+                                                false );
+
+  EXPECT_THAT( completer.CandidatesForQueryAndType( "oo", "foo" ),
+               ElementsAre( "foo",
+                            "foogoo" ) );
+}
+
 TEST( IdentifierCompleterTest, TagsEndToEndWorks ) {
   IdentifierCompleter completer;
   std::vector< std::string > tag_files;

cpp/ycm/tests/main.cpp

@@ -0,0 +1,13 @@
+#include "gtest/gtest.h"
+#include "gmock/gmock.h"
+#include <boost/python.hpp>
+
+int main( int argc, char **argv ) {
+  Py_Initialize();
+  // Necessary because of usage of the ReleaseGil class
+  PyEval_InitThreads();
+
+  testing::InitGoogleMock(&argc, argv);
+  return RUN_ALL_TESTS();
+}
+

cpp/ycm/tests/testdata/basic.cpp

@@ -2,7 +2,7 @@ struct Foo {
   int x;
   int y;
   char c;
-}
+};
 
 int main()
 {

cpp/ycm/versioning.cpp

@@ -0,0 +1,26 @@
+// Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+//
+// This file is part of YouCompleteMe.
+//
+// YouCompleteMe is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// YouCompleteMe is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace YouCompleteMe {
+
+int YcmCoreVersion() {
+  // We increment this every time when we want to force users to recompile
+  // ycm_core.
+  return 7;
+}
+
+}  // namespace YouCompleteMe

cpp/ycm/versioning.h

@@ -0,0 +1,22 @@
+// Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+//
+// This file is part of YouCompleteMe.
+//
+// YouCompleteMe is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// YouCompleteMe is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace YouCompleteMe {
+
+int YcmCoreVersion();
+
+}  // namespace YouCompleteMe

cpp/ycm/ycm_client_support.cpp

@@ -0,0 +1,44 @@
+// Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+//
+// This file is part of YouCompleteMe.
+//
+// YouCompleteMe is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// YouCompleteMe is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+#include "IdentifierCompleter.h"
+#include "PythonSupport.h"
+#include "versioning.h"
+
+#include <boost/python.hpp>
+#include <boost/utility.hpp>
+
+
+BOOST_PYTHON_MODULE(ycm_client_support)
+{
+  using namespace boost::python;
+  using namespace YouCompleteMe;
+
+  // Necessary because of usage of the ReleaseGil class
+  PyEval_InitThreads();
+
+  def( "FilterAndSortCandidates", FilterAndSortCandidates );
+  def( "YcmCoreVersion", YcmCoreVersion );
+}
+
+// Boost.Thread forces us to implement this.
+// We don't use any thread-specific (local) storage so it's fine to implement
+// this as an empty function.
+namespace boost {
+void tss_cleanup_implemented() {}
+};
+

cpp/ycm/ycm_core.cpp

@@ -17,7 +17,7 @@
 
 #include "IdentifierCompleter.h"
 #include "PythonSupport.h"
-#include "Future.h"
+#include "versioning.h"
 
 #ifdef USE_CLANG_COMPLETER
 #  include "ClangCompleter.h"
@@ -33,8 +33,7 @@
 #include <boost/utility.hpp>
 #include <boost/python/suite/indexing/vector_indexing_suite.hpp>
 
-bool HasClangSupport()
+bool HasClangSupport() {
-{
 #ifdef USE_CLANG_COMPLETER
   return true;
 #else
@@ -42,61 +41,37 @@ bool HasClangSupport()
 #endif // USE_CLANG_COMPLETER
 }
 
-int YcmCoreVersion()
-{
-  // We increment this every time when we want to force users to recompile
-  // ycm_core.
-  return 4;
-}
-
 
 BOOST_PYTHON_MODULE(ycm_core)
 {
   using namespace boost::python;
   using namespace YouCompleteMe;
 
+  // Necessary because of usage of the ReleaseGil class
+  PyEval_InitThreads();
+
   def( "HasClangSupport", HasClangSupport );
   def( "FilterAndSortCandidates", FilterAndSortCandidates );
   def( "YcmCoreVersion", YcmCoreVersion );
 
   class_< IdentifierCompleter, boost::noncopyable >( "IdentifierCompleter" )
-    .def( "EnableThreading", &IdentifierCompleter::EnableThreading )
     .def( "AddIdentifiersToDatabase",
           &IdentifierCompleter::AddIdentifiersToDatabase )
-    .def( "AddIdentifiersToDatabaseFromTagFilesAsync",
+    .def( "AddIdentifiersToDatabaseFromTagFiles",
-          &IdentifierCompleter::AddIdentifiersToDatabaseFromTagFilesAsync )
+          &IdentifierCompleter::AddIdentifiersToDatabaseFromTagFiles )
-    .def( "AddIdentifiersToDatabaseFromBufferAsync",
+    .def( "AddIdentifiersToDatabaseFromBuffer",
-          &IdentifierCompleter::AddIdentifiersToDatabaseFromBufferAsync )
+          &IdentifierCompleter::AddIdentifiersToDatabaseFromBuffer )
-    .def( "CandidatesForQueryAndTypeAsync",
+    .def( "CandidatesForQueryAndType",
-          &IdentifierCompleter::CandidatesForQueryAndTypeAsync );
+          &IdentifierCompleter::CandidatesForQueryAndType );
 
   // TODO: rename these *Vec classes to *Vector; don't forget the python file
   class_< std::vector< std::string >,
       boost::shared_ptr< std::vector< std::string > > >( "StringVec" )
     .def( vector_indexing_suite< std::vector< std::string > >() );
 
-  class_< Future< AsyncResults > >( "FutureResults" )
-    .def( "ResultsReady", &Future< AsyncResults >::ResultsReady )
-    .def( "GetResults", &Future< AsyncResults >::GetResults );
-
-  class_< Future< void > >( "FutureVoid" )
-    .def( "ResultsReady", &Future< void >::ResultsReady )
-    .def( "GetResults", &Future< void >::GetResults );
-
 #ifdef USE_CLANG_COMPLETER
   def( "ClangVersion", ClangVersion );
 
-  class_< Future< AsyncCompletions > >( "FutureCompletions" )
-    .def( "ResultsReady", &Future< AsyncCompletions >::ResultsReady )
-    .def( "GetResults", &Future< AsyncCompletions >::GetResults );
-
-  class_< Future< AsyncCompilationInfoForFile > >(
-      "FutureCompilationInfoForFile" )
-    .def( "ResultsReady",
-          &Future< AsyncCompilationInfoForFile >::ResultsReady )
-    .def( "GetResults",
-          &Future< AsyncCompilationInfoForFile >::GetResults );
-
   // CAREFUL HERE! For filename and contents we are referring directly to
   // Python-allocated and -managed memory since we are accepting pointers to
   // data members of python objects. We need to ensure that those objects
@@ -116,17 +91,13 @@ BOOST_PYTHON_MODULE(ycm_core)
     .def( vector_indexing_suite< std::vector< UnsavedFile > >() );
 
   class_< ClangCompleter, boost::noncopyable >( "ClangCompleter" )
-    .def( "EnableThreading", &ClangCompleter::EnableThreading )
-    .def( "DiagnosticsForFile", &ClangCompleter::DiagnosticsForFile )
     .def( "GetDeclarationLocation", &ClangCompleter::GetDeclarationLocation )
     .def( "GetDefinitionLocation", &ClangCompleter::GetDefinitionLocation )
-    .def( "DeleteCachesForFileAsync",
+    .def( "DeleteCachesForFile", &ClangCompleter::DeleteCachesForFile )
-          &ClangCompleter::DeleteCachesForFileAsync )
     .def( "UpdatingTranslationUnit", &ClangCompleter::UpdatingTranslationUnit )
-    .def( "UpdateTranslationUnitAsync",
+    .def( "UpdateTranslationUnit", &ClangCompleter::UpdateTranslationUnit )
-          &ClangCompleter::UpdateTranslationUnitAsync )
+    .def( "CandidatesForLocationInFile",
-    .def( "CandidatesForQueryAndLocationInFileAsync",
+          &ClangCompleter::CandidatesForLocationInFile );
-          &ClangCompleter::CandidatesForQueryAndLocationInFileAsync );
 
   class_< CompletionData >( "CompletionData" )
     .def( "TextToInsertInBuffer", &CompletionData::TextToInsertInBuffer )
@@ -160,13 +131,12 @@ BOOST_PYTHON_MODULE(ycm_core)
 
   class_< CompilationDatabase, boost::noncopyable >(
       "CompilationDatabase", init< std::string >() )
-    .def( "EnableThreading", &CompilationDatabase::EnableThreading )
     .def( "DatabaseSuccessfullyLoaded",
           &CompilationDatabase::DatabaseSuccessfullyLoaded )
+    .def( "AlreadyGettingFlags",
+          &CompilationDatabase::AlreadyGettingFlags )
     .def( "GetCompilationInfoForFile",
-          &CompilationDatabase::GetCompilationInfoForFile )
+          &CompilationDatabase::GetCompilationInfoForFile );
-    .def( "GetCompilationInfoForFileAsync",
-          &CompilationDatabase::GetCompilationInfoForFileAsync );
 
   class_< CompilationInfoForFile,
       boost::shared_ptr< CompilationInfoForFile > >(

doc/youcompleteme.txt

@@ -1,4 +1,4 @@
-*youcompleteme*  YouCompleteMe: a code-completion engine for Vim
+*youcompleteme.txt*  YouCompleteMe: a code-completion engine for Vim
 
 ===============================================================================
 Contents ~
@@ -10,20 +10,22 @@ Contents ~
  5. Full Installation Guide             |youcompleteme-full-installation-guide|
  6. User Guide                                       |youcompleteme-user-guide|
   1. General Usage                                |youcompleteme-general-usage|
-  2. Completion string ranking        |youcompleteme-completion-string-ranking|
+  2. Client-server architecture      |youcompleteme-client-server-architecture|
-  3. General Semantic Completion Engine Usage |youcompleteme-general-semantic-completion-engine-usage|
+  3. Completion string ranking        |youcompleteme-completion-string-ranking|
-  4. C-family Semantic Completion Engine Usage |youcompleteme-c-family-semantic-completion-engine-usage|
+  4. General Semantic Completion Engine Usage |youcompleteme-general-semantic-completion-engine-usage|
-  5. Python semantic completion      |youcompleteme-python-semantic-completion|
+  5. C-family Semantic Completion Engine Usage |youcompleteme-c-family-semantic-completion-engine-usage|
-  6. C# semantic completion               |youcompleteme-c-semantic-completion|
+  6. Python semantic completion      |youcompleteme-python-semantic-completion|
-  7. Semantic completion for other languages |youcompleteme-semantic-completion-for-other-languages|
+  7. C# semantic completion               |youcompleteme-c-semantic-completion|
-  8. Writing New Semantic Completers |youcompleteme-writing-new-semantic-completers|
+  8. Semantic completion for other languages |youcompleteme-semantic-completion-for-other-languages|
-  9. Syntastic integration                |youcompleteme-syntastic-integration|
+  9. Writing New Semantic Completers |youcompleteme-writing-new-semantic-completers|
+  10. Syntastic integration               |youcompleteme-syntastic-integration|
  7. Commands                                           |youcompleteme-commands|
-  1. The |:YcmForceCompileAndDiagnostics| command
+  1. The |:YcmRestartServer| command
-  2. The |:YcmDiags| command
+  2. The |:YcmForceCompileAndDiagnostics| command
-  3. The |:YcmShowDetailedDiagnostic| command
+  3. The |:YcmDiags| command
-  4. The |:YcmDebugInfo| command
+  4. The |:YcmShowDetailedDiagnostic| command
-  5. The |:YcmCompleter| command
+  5. The |:YcmDebugInfo| command
+  6. The |:YcmCompleter| command
  8. YcmCompleter subcommands           |youcompleteme-ycmcompleter-subcommands|
   1. The |GoToDeclaration| subcommand
   2. The |GoToDefinition| subcommand
@@ -45,23 +47,27 @@ Contents ~
   10. The |g:ycm_collect_identifiers_from_comments_and_strings| option
   11. The |g:ycm_collect_identifiers_from_tags_files| option
   12. The |g:ycm_seed_identifiers_with_syntax| option
-  13. The |g:ycm_csharp_server_port| option
+  13. The |g:ycm_server_use_vim_stdout| option
-  14. The |g:ycm_auto_start_csharp_server| option
+  14. The |g:ycm_server_keep_logfiles| option
-  15. The |g:ycm_auto_stop_csharp_server| option
+  15. The |g:ycm_server_log_level| option
-  16. The |g:ycm_add_preview_to_completeopt| option
+  16. The |g:ycm_server_idle_suicide_seconds| option
-  17. The |g:ycm_autoclose_preview_window_after_completion| option
+  17. The |g:ycm_csharp_server_port| option
-  18. The |g:ycm_autoclose_preview_window_after_insertion| option
+  18. The |g:ycm_auto_start_csharp_server| option
-  19. The |g:ycm_max_diagnostics_to_display| option
+  19. The |g:ycm_auto_stop_csharp_server| option
-  20. The |g:ycm_key_list_select_completion| option
+  20. The |g:ycm_add_preview_to_completeopt| option
-  21. The |g:ycm_key_list_previous_completion| option
+  21. The |g:ycm_autoclose_preview_window_after_completion| option
-  22. The |g:ycm_key_invoke_completion| option
+  22. The |g:ycm_autoclose_preview_window_after_insertion| option
-  23. The |g:ycm_key_detailed_diagnostics| option
+  23. The |g:ycm_max_diagnostics_to_display| option
-  24. The |g:ycm_global_ycm_extra_conf| option
+  24. The |g:ycm_key_list_select_completion| option
-  25. The |g:ycm_confirm_extra_conf| option
+  25. The |g:ycm_key_list_previous_completion| option
-  26. The |g:ycm_extra_conf_globlist| option
+  26. The |g:ycm_key_invoke_completion| option
-  27. The |g:ycm_filepath_completion_use_working_dir| option
+  27. The |g:ycm_key_detailed_diagnostics| option
-  28. The |g:ycm_semantic_triggers| option
+  28. The |g:ycm_global_ycm_extra_conf| option
-  29. The |g:ycm_cache_omnifunc| option
+  29. The |g:ycm_confirm_extra_conf| option
+  30. The |g:ycm_extra_conf_globlist| option
+  31. The |g:ycm_filepath_completion_use_working_dir| option
+  32. The |g:ycm_semantic_triggers| option
+  33. The |g:ycm_cache_omnifunc| option
  10. FAQ                                                    |youcompleteme-faq|
   1. I get a linker warning regarding |libpython| on Mac when compiling YCM
   2. I get a weird window at the top of my file when I use the semantic engine |youcompleteme-i-get-weird-window-at-top-of-my-file-when-i-use-semantic-engine|
@@ -182,8 +188,9 @@ local binary folder (for example '/usr/local/bin/mvim') and then symlink it:
 Install YouCompleteMe with Vundle [11].
 
 **Remember:** YCM is a plugin with a compiled component. If you **update** YCM
-using Vundle and the ycm_core library API has changed (happens rarely), YCM
+using Vundle and the ycm_support_libs library APIs have changed (happens
-will notify you to recompile it. You should then rerun the install process.
+rarely), YCM will notify you to recompile it. You should then rerun the install
+process.
 
 It's recommended that you have the latest Xcode installed along with the latest
 Command Line Tools (that you install from within Xcode).
@@ -230,8 +237,9 @@ from source [14] (don't worry, it's easy).
 Install YouCompleteMe with Vundle [11].
 
 **Remember:** YCM is a plugin with a compiled component. If you **update** YCM
-using Vundle and the ycm_core library API has changed (happens rarely), YCM
+using Vundle and the ycm_support_libs library APIs have changed (happens
-will notify you to recompile it. You should then rerun the install process.
+rarely), YCM will notify you to recompile it. You should then rerun the install
+process.
 
 Install development tools and CMake: 'sudo apt-get install build-essential
 cmake'
@@ -281,8 +289,9 @@ that platform).
 See the _FAQ_ if you have any issues.
 
 **Remember:** YCM is a plugin with a compiled component. If you **update** YCM
-using Vundle and the ycm_core library API has changed (happens rarely), YCM
+using Vundle and the ycm_support_libs library APIs have changed (happens
-will notify you to recompile it. You should then rerun the install process.
+rarely), YCM will notify you to recompile it. You should then rerun the install
+process.
 
 **Please follow the instructions carefully. Read EVERY WORD.**
 
@@ -319,8 +328,8 @@ will notify you to recompile it. You should then rerun the install process.
    official binaries from llvm.org [18] if at all possible. Make sure you
    download the correct archive file for your OS.
 
-4. **Compile the 'ycm_core' plugin plugin** (ha!) that YCM needs. This is
+4. **Compile the 'ycm_support_libs' libraries** that YCM needs. These libs
-   the C++ engine that YCM uses to get fast completions.
+   are the C++ engines that YCM uses to get fast completions.
 
    You will need to have 'cmake' installed in order to generate the required
    makefiles. Linux users can install cmake with their package manager
@@ -359,7 +368,7 @@ will notify you to recompile it. You should then rerun the install process.
 <
    Now that makefiles have been generated, simply run:
 >
-   make ycm_core
+   make ycm_support_libs
 <
    For those who want to use the system version of libclang, you would pass
    '-DUSE_SYSTEM_LIBCLANG=ON' to cmake _instead of_ the
@@ -427,6 +436,15 @@ YCM automatically detects which completion engine would be the best in any
 situation. On occasion, it queries several of them at once, merges the outputs
 and presents the results to you.
 
+-------------------------------------------------------------------------------
+                                     *youcompleteme-client-server-architecture*
+Client-server architecture ~
+
+YCM has a client-server architecture; the Vim part of YCM is only a thin client
+that talks to the 'ycmd' HTTP+JSON server that has the vast majority of YCM
+logic and functionality. The server is started and stopped automatically as you
+start and stop Vim.
+
 -------------------------------------------------------------------------------
                                       *youcompleteme-completion-string-ranking*
 Completion string ranking ~
@@ -624,6 +642,12 @@ yours truly.
                                                        *youcompleteme-commands*
 Commands ~
 
+-------------------------------------------------------------------------------
+The *:YcmRestartServer* command
+
+If the 'ycmd' completion server suddenly stops for some reason, you can restart
+it with this command.
+
 -------------------------------------------------------------------------------
 The *:YcmForceCompileAndDiagnostics* command
 
@@ -694,7 +718,7 @@ The *GoToDeclaration* subcommand
 
 Looks up the symbol under the cursor and jumps to its declaration.
 
-Supported in filetypes: 'c, cpp, objc, objcpp, python'
+Supported in filetypes: 'c, cpp, objc, objcpp, python, cs'
 
 -------------------------------------------------------------------------------
 The *GoToDefinition* subcommand
@@ -706,7 +730,7 @@ when the definition of the symbol is in the current translation unit. A
 translation unit consists of the file you are editing and all the files you are
 including with '#include' directives (directly or indirectly) in that file.
 
-Supported in filetypes: 'c, cpp, objc, objcpp, python'
+Supported in filetypes: 'c, cpp, objc, objcpp, python, cs'
 
 -------------------------------------------------------------------------------
 The *GoToDefinitionElseDeclaration* subcommand
@@ -715,7 +739,7 @@ Looks up the symbol under the cursor and jumps to its definition if possible;
 if the definition is not accessible from the current translation unit, jumps to
 the symbol's declaration.
 
-Supported in filetypes: 'c, cpp, objc, objcpp, python'
+Supported in filetypes: 'c, cpp, objc, objcpp, python, cs'
 
 -------------------------------------------------------------------------------
 The *ClearCompilationFlagCache* subcommand
@@ -982,6 +1006,64 @@ Default: '0'
   let g:ycm_seed_identifiers_with_syntax = 0
 <
 -------------------------------------------------------------------------------
+The *g:ycm_server_use_vim_stdout* option
+
+By default, the 'ycmd' completion server writes logs to logfiles. When this
+option is set to '1', the server writes logs to Vim's stdout (so you'll see
+them in the console).
+
+Default: '0'
+>
+  let g:ycm_server_use_vim_stdout = 0
+<
+-------------------------------------------------------------------------------
+The *g:ycm_server_keep_logfiles* option
+
+When this option is set to '1', the 'ycmd' completion server will keep the
+logfiles around after shutting down (they are deleted on shutdown by default).
+
+To see where the logfiles are, call |:YcmDebugInfo|.
+
+Default: '0'
+>
+  let g:ycm_server_keep_logfiles = 0
+<
+-------------------------------------------------------------------------------
+The *g:ycm_server_log_level* option
+
+The logging level that the 'ycmd' completion server uses. Valid values are the
+following, from most verbose to least verbose: - 'debug' - 'info' - 'warning' -
+'error' - 'critical'
+
+Note that 'debug' is _very_ verbose.
+
+Default: 'info'
+>
+  let g:ycm_server_log_level = 'info'
+<
+-------------------------------------------------------------------------------
+The *g:ycm_server_idle_suicide_seconds* option
+
+This option sets the number of seconds of 'ycmd' server idleness (no requests
+received) after which the server stops itself. NOTE: the YCM Vim client sends a
+shutdown request to the server when Vim is shutting down.
+
+If your Vim crashes for instance, 'ycmd' never gets the shutdown command and
+becomes a zombie process. This option prevents such zombies from sticking
+around forever.
+
+The default option is '43200' seconds which is 12 hours. The reason for the
+interval being this long is to prevent the server from shutting down if you
+leave your computer (and Vim) turned on during the night.
+
+The server "heartbeat" that checks whether this interval has passed occurs
+every 10 minutes.
+
+Default: '43200'
+>
+  let g:ycm_server_idle_suicide_seconds = 43200
+<
+-------------------------------------------------------------------------------
 The *g:ycm_csharp_server_port* option
 
 The port number (on 'localhost') on which the OmniSharp server should be
@@ -1575,6 +1657,8 @@ License ~
 This software is licensed under the GPL v3 license [31]. © 2012 Strahinja Val
 Markovic <val@markovic.io>.
 
+  Image: Bitdeli Badge [32]
+
 ===============================================================================
                                                      *youcompleteme-references*
 References ~
@@ -1610,5 +1694,7 @@ References ~
 [29] https://groups.google.com/forum/?hl=en#!forum/ycm-users
 [30] https://github.com/Valloric/YouCompleteMe/issues?state=open
 [31] http://www.gnu.org/copyleft/gpl.html
+[32] https://bitdeli.com/free
+[33] https://d2weczhvl823v0.cloudfront.net/Valloric/youcompleteme/trend.png
 
 vim: ft=help

install.sh

@@ -77,7 +77,7 @@ function install {
     cmake -G "Unix Makefiles" "$@" . $ycm_dir/cpp
   fi
 
-  make -j $(num_cores) ycm_core
+  make -j $(num_cores) ycm_support_libs
   popd
   rm -rf $build_dir
 }
@@ -106,6 +106,22 @@ function usage {
   exit 0
 }
 
+function check_third_party_libs {
+  libs_present=true
+  for folder in third_party/*; do
+    num_files_in_folder=$(find $folder -maxdepth 1 -mindepth 1 | wc -l)
+    if [[ $num_files_in_folder -eq 0 ]]; then
+      libs_present=false
+    fi
+  done
+
+  if ! $libs_present; then
+    echo "Some folders in ./third_party are empty; you probably forgot to run:"
+    printf "\n\tgit submodule update --init --recursive\n\n"
+    exit 1
+  fi
+}
+
 cmake_args=""
 omnisharp_completer=false
 for flag in $@; do
@@ -130,6 +146,8 @@ if [[ $cmake_args == *-DUSE_SYSTEM_LIBCLANG=ON* ]] && \
   usage
 fi
 
+check_third_party_libs
+
 if ! command_exists cmake; then
   echo "CMake is required to build YouCompleteMe."
   cmake_install
@@ -138,7 +156,7 @@ fi
 if [ -z "$YCM_TESTRUN" ]; then
   install $cmake_args $EXTRA_CMAKE_ARGS
 else
-  testrun $cmake_args -DUSE_DEV_FLAGS=ON $EXTRA_CMAKE_ARGS
+  testrun $cmake_args $EXTRA_CMAKE_ARGS
 fi
 
 if $omnisharp_completer; then

plugin/youcompleteme.vim

@@ -37,11 +37,14 @@ let s:script_folder_path = escape( expand( '<sfile>:p:h' ), '\' )
 
 function! s:HasYcmCore()
   let path_prefix = s:script_folder_path . '/../python/'
-  if filereadable(path_prefix . 'ycm_core.so')
+  if filereadable(path_prefix . 'ycm_client_support.so') &&
+        \ filereadable(path_prefix . 'ycm_core.so')
     return 1
-  elseif filereadable(path_prefix . 'ycm_core.pyd')
+  elseif filereadable(path_prefix . 'ycm_client_support.pyd') &&
+        \ filereadable(path_prefix . 'ycm_core.pyd')
     return 1
-  elseif filereadable(path_prefix . 'ycm_core.dll')
+  elseif filereadable(path_prefix . 'ycm_client_support.dll') &&
+        \ filereadable(path_prefix . 'ycm_core.dll')
     return 1
   endif
   return 0
@@ -52,7 +55,8 @@ let g:ycm_check_if_ycm_core_present =
 
 if g:ycm_check_if_ycm_core_present && !s:HasYcmCore()
   echohl WarningMsg |
-        \ echomsg "ycm_core.[so|pyd|dll] not detected; you need to compile " .
+        \ echomsg "ycm_client_support.[so|pyd|dll] and " .
+        \ "ycm_core.[so|pyd|dll] not detected; you need to compile " .
         \ "YCM before using it. Read the docs!" |
         \ echohl None
   finish
@@ -60,31 +64,11 @@ endif
 
 let g:loaded_youcompleteme = 1
 
-let g:ycm_min_num_of_chars_for_completion  =
+" NOTE: Most defaults are in default_settings.json. They are loaded into Vim
-      \ get( g:, 'ycm_min_num_of_chars_for_completion', 2 )
+" global with the 'ycm_' prefix if such a key does not already exist; thus, the
-
+" user can override the defaults.
-let g:ycm_min_num_identifier_candidate_chars =
+" The only defaults that are here are the ones that are only relevant to the YCM
-      \ get( g:, 'ycm_min_num_identifier_candidate_chars', 0 )
+" Vim client and not the server.
-
-let g:ycm_filetype_whitelist =
-      \ get( g:, 'ycm_filetype_whitelist', {
-      \   '*' : 1,
-      \ } )
-
-" The fallback to g:ycm_filetypes_to_completely_ignore is here because of
-" backwards compatibility with previous versions of YCM.
-let g:ycm_filetype_blacklist =
-      \ get( g:, 'ycm_filetype_blacklist',
-      \   get( g:, 'ycm_filetypes_to_completely_ignore', {
-      \     'notes' : 1,
-      \     'markdown' : 1,
-      \     'text' : 1,
-      \     'unite' : 1,
-      \     'tagbar' : 1,
-      \ } ) )
-
-let g:ycm_filetype_specific_completion_to_disable =
-      \ get( g:, 'ycm_filetype_specific_completion_to_disable', {} )
 
 let g:ycm_register_as_syntastic_checker =
       \ get( g:, 'ycm_register_as_syntastic_checker', 1 )
@@ -95,30 +79,12 @@ let g:ycm_allow_changing_updatetime =
 let g:ycm_add_preview_to_completeopt =
       \ get( g:, 'ycm_add_preview_to_completeopt', 0 )
 
-let g:ycm_complete_in_comments =
-      \ get( g:, 'ycm_complete_in_comments', 0 )
-
-let g:ycm_complete_in_strings =
-      \ get( g:, 'ycm_complete_in_strings', 1 )
-
-let g:ycm_collect_identifiers_from_comments_and_strings =
-      \ get( g:, 'ycm_collect_identifiers_from_comments_and_strings', 0 )
-
-let g:ycm_collect_identifiers_from_tags_files =
-      \ get( g:, 'ycm_collect_identifiers_from_tags_files', 0 )
-
-let g:ycm_seed_identifiers_with_syntax =
-      \ get( g:, 'ycm_seed_identifiers_with_syntax', 0 )
-
 let g:ycm_autoclose_preview_window_after_completion =
       \ get( g:, 'ycm_autoclose_preview_window_after_completion', 0 )
 
 let g:ycm_autoclose_preview_window_after_insertion =
       \ get( g:, 'ycm_autoclose_preview_window_after_insertion', 0 )
 
-let g:ycm_max_diagnostics_to_display =
-      \ get( g:, 'ycm_max_diagnostics_to_display', 30 )
-
 let g:ycm_key_list_select_completion =
       \ get( g:, 'ycm_key_list_select_completion', ['<TAB>', '<Down>'] )
 
@@ -131,34 +97,21 @@ let g:ycm_key_invoke_completion =
 let g:ycm_key_detailed_diagnostics =
       \ get( g:, 'ycm_key_detailed_diagnostics', '<leader>d' )
 
-let g:ycm_global_ycm_extra_conf =
-      \ get( g:, 'ycm_global_ycm_extra_conf', '' )
-
-let g:ycm_confirm_extra_conf =
-      \ get( g:, 'ycm_confirm_extra_conf', 1 )
-
-let g:ycm_extra_conf_globlist =
-      \ get( g:, 'ycm_extra_conf_globlist', [] )
-
-let g:ycm_filepath_completion_use_working_dir =
-      \ get( g:, 'ycm_filepath_completion_use_working_dir', 0 )
-
-" Default semantic triggers are in python/ycm/completers/completer.py, these
-" just append new triggers to the default dict.
-let g:ycm_semantic_triggers =
-      \ get( g:, 'ycm_semantic_triggers', {} )
-
 let g:ycm_cache_omnifunc =
       \ get( g:, 'ycm_cache_omnifunc', 1 )
 
-let g:ycm_auto_start_csharp_server =
+let g:ycm_server_use_vim_stdout =
-      \ get( g:, 'ycm_auto_start_csharp_server', 1 )
+      \ get( g:, 'ycm_server_use_vim_stdout', 0 )
 
-let g:ycm_auto_stop_csharp_server =
+let g:ycm_server_log_level =
-            \ get( g:, 'ycm_auto_stop_csharp_server', 1 )
+      \ get( g:, 'ycm_server_log_level', 'info' )
+
+let g:ycm_server_keep_logfiles =
+      \ get( g:, 'ycm_server_keep_logfiles', 0 )
+
+let g:ycm_server_idle_suicide_seconds =
+      \ get( g:, 'ycm_server_idle_suicide_seconds', 43200 )
 
-let g:ycm_csharp_server_port =
-      \ get( g:, 'ycm_csharp_server_port', 2000 )
 
 " On-demand loading. Let's use the autoload folder and not slow down vim's
 " startup procedure.

print_todos.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+ag \
+--ignore gmock \
+--ignore jedi/ \
+--ignore OmniSharpServer \
+--ignore testdata \
+TODO \
+cpp/ycm python autoload plugin

python/test_requirements.txt

@@ -1,3 +1,6 @@
 flake8>=2.0
 mock>=1.0.1
 nose>=1.3.0
+PyHamcrest>=1.7.2
+WebTest>=2.0.9
+

python/ycm/base.py

@@ -17,21 +17,42 @@
 # You should have received a copy of the GNU General Public License
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
 import re
 import vim
 from ycm import vimsupport
 from ycm import utils
+from ycm import user_options_store
+import ycm_client_support
 
-try:
+YCM_VAR_PREFIX = 'ycm_'
-  import ycm_core
+
-except ImportError as e:
+
-  vimsupport.PostVimMessage(
+def BuildServerConf():
-    'Error importing ycm_core. Are you sure you have placed a version 3.2+ '
+  """Builds a dictionary mapping YCM Vim user options to values. Option names
-    'libclang.[so|dll|dylib] in folder "{0}"? See the Installation Guide in '
+  don't have the 'ycm_' prefix."""
-    'the docs. Full error: {1}'.format(
+
-      os.path.dirname( os.path.dirname( os.path.abspath( __file__ ) ) ),
+  vim_globals = vimsupport.GetReadOnlyVimGlobals( force_python_objects = True )
-      str( e ) ) )
+  server_conf = {}
+  for key, value in vim_globals.items():
+    if not key.startswith( YCM_VAR_PREFIX ):
+      continue
+    try:
+      new_value = int( value )
+    except:
+      new_value = value
+    new_key = key[ len( YCM_VAR_PREFIX ): ]
+    server_conf[ new_key ] = new_value
+
+  return server_conf
+
+
+def LoadJsonDefaultsIntoVim():
+  defaults = user_options_store.DefaultOptions()
+  vim_defaults = {}
+  for key, value in defaults.iteritems():
+    vim_defaults[ 'ycm_' + key ] = value
+
+  vimsupport.LoadDictIntoVimGlobals( vim_defaults, overwrite = False )
 
 
 def CompletionStartColumn():
@@ -121,11 +142,11 @@ def AdjustCandidateInsertionText( candidates ):
   return new_candidates
 
 
-COMPATIBLE_WITH_CORE_VERSION = 4
+COMPATIBLE_WITH_CORE_VERSION = 7
 
 def CompatibleWithYcmCore():
   try:
-    current_core_version = ycm_core.YcmCoreVersion()
+    current_core_version = ycm_client_support.YcmCoreVersion()
   except AttributeError:
     return False
 

python/ycm/client/base_request.py

@@ -0,0 +1,154 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+import vim
+import json
+import requests
+import urlparse
+from retries import retries
+from requests_futures.sessions import FuturesSession
+from concurrent.futures import ThreadPoolExecutor
+from ycm import vimsupport
+from ycm.server.responses import ServerError, UnknownExtraConf
+
+HEADERS = {'content-type': 'application/json'}
+EXECUTOR = ThreadPoolExecutor( max_workers = 10 )
+# Setting this to None seems to screw up the Requests/urllib3 libs.
+DEFAULT_TIMEOUT_SEC = 30
+
+class BaseRequest( object ):
+  def __init__( self ):
+    pass
+
+
+  def Start( self ):
+    pass
+
+
+  def Done( self ):
+    return True
+
+
+  def Response( self ):
+    return {}
+
+
+  # This is the blocking version of the method. See below for async.
+  # |timeout| is num seconds to tolerate no response from server before giving
+  # up; see Requests docs for details (we just pass the param along).
+  @staticmethod
+  def PostDataToHandler( data, handler, timeout = DEFAULT_TIMEOUT_SEC ):
+    return JsonFromFuture( BaseRequest.PostDataToHandlerAsync( data,
+                                                               handler,
+                                                               timeout ) )
+
+
+  # This returns a future! Use JsonFromFuture to get the value.
+  # |timeout| is num seconds to tolerate no response from server before giving
+  # up; see Requests docs for details (we just pass the param along).
+  @staticmethod
+  def PostDataToHandlerAsync( data, handler, timeout = DEFAULT_TIMEOUT_SEC ):
+    def PostData( data, handler, timeout ):
+      return BaseRequest.session.post( _BuildUri( handler ),
+                                       data = json.dumps( data ),
+                                       headers = HEADERS,
+                                       timeout = timeout )
+
+    @retries( 5, delay = 0.5, backoff = 1.5 )
+    def DelayedPostData( data, handler ):
+      return requests.post( _BuildUri( handler ),
+                            data = json.dumps( data ),
+                            headers = HEADERS )
+
+    if not _CheckServerIsHealthyWithCache():
+      return EXECUTOR.submit( DelayedPostData, data, handler )
+
+    return PostData( data, handler, timeout )
+
+
+  session = FuturesSession( executor = EXECUTOR )
+  server_location = 'http://localhost:6666'
+
+
+def BuildRequestData( start_column = None,
+                      query = None,
+                      include_buffer_data = True ):
+  line, column = vimsupport.CurrentLineAndColumn()
+  filepath = vimsupport.GetCurrentBufferFilepath()
+  request_data = {
+    'filetypes': vimsupport.CurrentFiletypes(),
+    'line_num': line,
+    'column_num': column,
+    'start_column': start_column,
+    'line_value': vim.current.line,
+    'filepath': filepath
+  }
+
+  if include_buffer_data:
+    request_data[ 'file_data' ] = vimsupport.GetUnsavedAndCurrentBufferData()
+  if query:
+    request_data[ 'query' ] = query
+
+  return request_data
+
+
+def JsonFromFuture( future ):
+  response = future.result()
+  if response.status_code == requests.codes.server_error:
+    _RaiseExceptionForData( response.json() )
+
+  # We let Requests handle the other status types, we only handle the 500
+  # error code.
+  response.raise_for_status()
+
+  if response.text:
+    return response.json()
+  return None
+
+
+def _BuildUri( handler ):
+  return urlparse.urljoin( BaseRequest.server_location, handler )
+
+
+SERVER_HEALTHY = False
+
+def _CheckServerIsHealthyWithCache():
+  global SERVER_HEALTHY
+
+  def _ServerIsHealthy():
+    response = requests.get( _BuildUri( 'healthy' ) )
+    response.raise_for_status()
+    return response.json()
+
+  if SERVER_HEALTHY:
+    return True
+
+  try:
+    SERVER_HEALTHY = _ServerIsHealthy()
+    return SERVER_HEALTHY
+  except:
+    return False
+
+
+def _RaiseExceptionForData( data ):
+  if data[ 'exception' ][ 'TYPE' ] == UnknownExtraConf.__name__:
+    raise UnknownExtraConf( data[ 'exception' ][ 'extra_conf_file' ] )
+
+  raise ServerError( '{0}: {1}'.format( data[ 'exception' ][ 'TYPE' ],
+                                        data[ 'message' ] ) )

python/ycm/client/command_request.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+import vim
+from ycm.client.base_request import BaseRequest, BuildRequestData, ServerError
+from ycm import vimsupport
+from ycm.utils import ToUtf8IfNeeded
+
+
+class CommandRequest( BaseRequest ):
+  def __init__( self, arguments, completer_target = None ):
+    super( CommandRequest, self ).__init__()
+    self._arguments = arguments
+    self._completer_target = ( completer_target if completer_target
+                               else 'filetype_default' )
+    self._is_goto_command = (
+        arguments and arguments[ 0 ].startswith( 'GoTo' ) )
+    self._response = None
+
+
+  def Start( self ):
+    request_data = BuildRequestData()
+    request_data.update( {
+      'completer_target': self._completer_target,
+      'command_arguments': self._arguments
+    } )
+    try:
+      self._response = self.PostDataToHandler( request_data,
+                                              'run_completer_command' )
+    except ServerError as e:
+      vimsupport.PostVimMessage( e )
+
+
+  def Response( self ):
+    return self._response
+
+
+  def RunPostCommandActionsIfNeeded( self ):
+    if not self._is_goto_command or not self.Done() or not self._response:
+      return
+
+    if isinstance( self._response, list ):
+      defs = [ _BuildQfListItem( x ) for x in self._response ]
+      vim.eval( 'setqflist( %s )' % repr( defs ) )
+      vim.eval( 'youcompleteme#OpenGoToList()' )
+    else:
+      vimsupport.JumpToLocation( self._response[ 'filepath' ],
+                                 self._response[ 'line_num' ] + 1,
+                                 self._response[ 'column_num' ] + 1)
+
+
+
+
+def SendCommandRequest( arguments, completer ):
+  request = CommandRequest( arguments, completer )
+  # This is a blocking call.
+  request.Start()
+  request.RunPostCommandActionsIfNeeded()
+  return request.Response()
+
+
+def _BuildQfListItem( goto_data_item ):
+  qf_item = {}
+  if 'filepath' in goto_data_item:
+    qf_item[ 'filename' ] = ToUtf8IfNeeded( goto_data_item[ 'filepath' ] )
+  if 'description' in goto_data_item:
+    qf_item[ 'text' ] = ToUtf8IfNeeded( goto_data_item[ 'description' ] )
+  if 'line_num' in goto_data_item:
+    qf_item[ 'lnum' ] = goto_data_item[ 'line_num' ] + 1
+  if 'column_num' in goto_data_item:
+    qf_item[ 'col' ] = goto_data_item[ 'column_num' ]
+  return qf_item

python/ycm/client/completion_request.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+from ycm import base
+from ycm import vimsupport
+from ycm.client.base_request import ( BaseRequest, BuildRequestData,
+                                      JsonFromFuture )
+
+TIMEOUT_SECONDS = 0.5
+
+class CompletionRequest( BaseRequest ):
+  def __init__( self, force_semantic = False ):
+    super( CompletionRequest, self ).__init__()
+
+    self._completion_start_column = base.CompletionStartColumn()
+
+    # This field is also used by the omni_completion_request subclass
+    self.request_data = BuildRequestData( self._completion_start_column )
+    if force_semantic:
+      self.request_data[ 'force_semantic' ] = True
+
+
+  def CompletionStartColumn( self ):
+    return self._completion_start_column
+
+
+  def Start( self, query ):
+    self.request_data[ 'query' ] = query
+    self._response_future = self.PostDataToHandlerAsync( self.request_data,
+                                                         'completions',
+                                                         TIMEOUT_SECONDS )
+
+
+  def Done( self ):
+    return self._response_future.done()
+
+
+  def Response( self ):
+    if not self._response_future:
+      return []
+    try:
+      return [ _ConvertCompletionDataToVimData( x )
+               for x in JsonFromFuture( self._response_future ) ]
+    except Exception as e:
+      vimsupport.PostVimMessage( str( e ) )
+    return []
+
+
+def _ConvertCompletionDataToVimData( completion_data ):
+  # see :h complete-items for a description of the dictionary fields
+  vim_data = {
+    'word' : completion_data[ 'insertion_text' ],
+    'dup'  : 1,
+  }
+
+  if 'menu_text' in completion_data:
+    vim_data[ 'abbr' ] = completion_data[ 'menu_text' ]
+  if 'extra_menu_info' in completion_data:
+    vim_data[ 'menu' ] = completion_data[ 'extra_menu_info' ]
+  if 'kind' in completion_data:
+    vim_data[ 'kind' ] = completion_data[ 'kind' ]
+  if 'detailed_info' in completion_data:
+    vim_data[ 'info' ] = completion_data[ 'detailed_info' ]
+
+  return vim_data

python/ycm/client/event_notification.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+from ycm import vimsupport
+from ycm.server.responses import UnknownExtraConf
+from ycm.client.base_request import ( BaseRequest, BuildRequestData,
+                                     JsonFromFuture )
+
+
+class EventNotification( BaseRequest ):
+  def __init__( self, event_name, extra_data = None ):
+    super( EventNotification, self ).__init__()
+    self._event_name = event_name
+    self._extra_data = extra_data
+    self._cached_response = None
+
+
+  def Start( self ):
+    request_data = BuildRequestData()
+    if self._extra_data:
+      request_data.update( self._extra_data )
+    request_data[ 'event_name' ] = self._event_name
+
+    self._response_future = self.PostDataToHandlerAsync( request_data,
+                                                         'event_notification' )
+
+
+  def Done( self ):
+    return self._response_future.done()
+
+
+  def Response( self ):
+    if self._cached_response:
+      return self._cached_response
+
+    if not self._response_future or self._event_name != 'FileReadyToParse':
+      return []
+
+    try:
+      try:
+        self._cached_response = JsonFromFuture( self._response_future )
+      except UnknownExtraConf as e:
+          if vimsupport.Confirm( str( e ) ):
+            _LoadExtraConfFile( e.extra_conf_file )
+    except Exception as e:
+      vimsupport.PostVimMessage( str( e ) )
+
+    if not self._cached_response:
+      return []
+
+    self._cached_response = [ _ConvertDiagnosticDataToVimData( x )
+                              for x in self._cached_response ]
+    return self._cached_response
+
+
+def _ConvertDiagnosticDataToVimData( diagnostic ):
+  # see :h getqflist for a description of the dictionary fields
+  # Note that, as usual, Vim is completely inconsistent about whether
+  # line/column numbers are 1 or 0 based in its various APIs. Here, it wants
+  # them to be 1-based.
+  return {
+    'bufnr' : vimsupport.GetBufferNumberForFilename( diagnostic[ 'filepath' ]),
+    'lnum'  : diagnostic[ 'line_num' ] + 1,
+    'col'   : diagnostic[ 'column_num' ] + 1,
+    'text'  : diagnostic[ 'text' ],
+    'type'  : diagnostic[ 'kind' ],
+    'valid' : 1
+  }
+
+
+def SendEventNotificationAsync( event_name, extra_data = None ):
+  event = EventNotification( event_name, extra_data )
+  event.Start()
+
+def _LoadExtraConfFile( filepath ):
+  BaseRequest.PostDataToHandler( { 'filepath': filepath },
+                                 'load_extra_conf_file' )

python/ycm/client/omni_completion_request.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+from ycm.client.completion_request import CompletionRequest
+
+
+class OmniCompletionRequest( CompletionRequest ):
+  def __init__( self, omni_completer ):
+    super( OmniCompletionRequest, self ).__init__()
+    self._omni_completer = omni_completer
+
+
+  def Start( self, query ):
+    self.request_data[ 'query' ] = query
+    self._results = self._omni_completer.ComputeCandidates( self.request_data )
+
+
+  def Done( self ):
+    return True
+
+
+  def Response( self ):
+    return self._results

python/ycm/completers/all/identifier_completer.py

@@ -18,175 +18,154 @@
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-import vim
+import logging
 import ycm_core
 from collections import defaultdict
 from ycm.completers.general_completer import GeneralCompleter
-from ycm.completers.general import syntax_parse
+# from ycm.completers.general import syntax_parse
-from ycm import vimsupport
 from ycm import utils
+from ycm.utils import ToUtf8IfNeeded
+from ycm.server import responses
 
 MAX_IDENTIFIER_COMPLETIONS_RETURNED = 10
-MIN_NUM_COMPLETION_START_CHARS = int( vimsupport.GetVariableValue(
-  "g:ycm_min_num_of_chars_for_completion" ) )
-MIN_NUM_CANDIDATE_SIZE_CHARS = int( vimsupport.GetVariableValue(
-  "g:ycm_min_num_identifier_candidate_chars" ) )
 SYNTAX_FILENAME = 'YCM_PLACEHOLDER_FOR_SYNTAX'
 
 
 class IdentifierCompleter( GeneralCompleter ):
-  def __init__( self ):
+  def __init__( self, user_options ):
-    super( IdentifierCompleter, self ).__init__()
+    super( IdentifierCompleter, self ).__init__( user_options )
-    self.completer = ycm_core.IdentifierCompleter()
+    self._completer = ycm_core.IdentifierCompleter()
-    self.completer.EnableThreading()
+    self._tags_file_last_mtime = defaultdict( int )
-    self.tags_file_last_mtime = defaultdict( int )
+    self._logger = logging.getLogger( __name__ )
-    self.filetypes_with_keywords_loaded = set()
 
 
-  def ShouldUseNow( self, start_column ):
+  def ShouldUseNow( self, request_data ):
-      return self.QueryLengthAboveMinThreshold( start_column )
+    return self.QueryLengthAboveMinThreshold( request_data )
 
 
-  def CandidatesForQueryAsync( self, query, unused_start_column ):
+  def ComputeCandidates( self, request_data ):
-    filetype = vim.eval( "&filetype" )
+    if not self.ShouldUseNow( request_data ):
-    self.completions_future = self.completer.CandidatesForQueryAndTypeAsync(
+      return []
-      utils.SanitizeQuery( query ),
+
-      filetype )
+    completions = self._completer.CandidatesForQueryAndType(
+      ToUtf8IfNeeded( utils.SanitizeQuery( request_data[ 'query' ] ) ),
+      ToUtf8IfNeeded( request_data[ 'filetypes' ][ 0 ] ) )
+
+    completions = completions[ : MAX_IDENTIFIER_COMPLETIONS_RETURNED ]
+    completions = _RemoveSmallCandidates(
+      completions, self.user_options[ 'min_num_identifier_candidate_chars' ] )
+
+    return [ responses.BuildCompletionData( x ) for x in completions ]
 
 
-  def AddIdentifier( self, identifier ):
+  def AddIdentifier( self, identifier, request_data ):
-    filetype = vim.eval( "&filetype" )
+    filetype = request_data[ 'filetypes' ][ 0 ]
-    filepath = vim.eval( "expand('%:p')" )
+    filepath = request_data[ 'filepath' ]
 
     if not filetype or not filepath or not identifier:
       return
 
     vector = ycm_core.StringVec()
-    vector.append( identifier )
+    vector.append( ToUtf8IfNeeded( identifier ) )
-    self.completer.AddIdentifiersToDatabase( vector,
+    self._logger.info( 'Adding ONE buffer identifier for file: %s', filepath )
-                                             filetype,
+    self._completer.AddIdentifiersToDatabase( vector,
-                                             filepath )
+                                             ToUtf8IfNeeded( filetype ),
+                                             ToUtf8IfNeeded( filepath ) )
 
 
-  def AddPreviousIdentifier( self ):
+  def AddPreviousIdentifier( self, request_data ):
-    self.AddIdentifier( PreviousIdentifier() )
+    self.AddIdentifier(
+      _PreviousIdentifier(
+        self.user_options[ 'min_num_of_chars_for_completion' ],
+        request_data ),
+      request_data )
 
 
-  def AddIdentifierUnderCursor( self ):
+  def AddIdentifierUnderCursor( self, request_data ):
-    cursor_identifier = vim.eval( 'expand("<cword>")' )
+    cursor_identifier = _GetCursorIdentifier( request_data )
     if not cursor_identifier:
       return
 
-    stripped_cursor_identifier = ''.join( ( x for x in
+    self.AddIdentifier( cursor_identifier, request_data )
-                                            cursor_identifier if
-                                            utils.IsIdentifierChar( x ) ) )
-    if not stripped_cursor_identifier:
-      return
-
-    self.AddIdentifier( stripped_cursor_identifier )
 
 
-  def AddBufferIdentifiers( self ):
+  def AddBufferIdentifiers( self, request_data ):
-    # TODO: use vimsupport.GetFiletypes; also elsewhere in file
+    filetype = request_data[ 'filetypes' ][ 0 ]
-    filetype = vim.eval( "&filetype" )
+    filepath = request_data[ 'filepath' ]
-    filepath = vim.eval( "expand('%:p')" )
+    collect_from_comments_and_strings = bool( self.user_options[
-    collect_from_comments_and_strings = vimsupport.GetBoolValue(
+      'collect_identifiers_from_comments_and_strings' ] )
-      "g:ycm_collect_identifiers_from_comments_and_strings" )
 
     if not filetype or not filepath:
       return
 
-    text = "\n".join( vim.current.buffer )
+    text = request_data[ 'file_data' ][ filepath ][ 'contents' ]
-    self.completer.AddIdentifiersToDatabaseFromBufferAsync(
+    self._logger.info( 'Adding buffer identifiers for file: %s', filepath )
-      text,
+    self._completer.AddIdentifiersToDatabaseFromBuffer(
-      filetype,
+      ToUtf8IfNeeded( text ),
-      filepath,
+      ToUtf8IfNeeded( filetype ),
+      ToUtf8IfNeeded( filepath ),
       collect_from_comments_and_strings )
 
 
-  def AddIdentifiersFromTagFiles( self ):
+  def AddIdentifiersFromTagFiles( self, tag_files ):
-    tag_files = vim.eval( 'tagfiles()' )
-    current_working_directory = os.getcwd()
     absolute_paths_to_tag_files = ycm_core.StringVec()
     for tag_file in tag_files:
-      absolute_tag_file = os.path.join( current_working_directory,
-                                        tag_file )
       try:
-        current_mtime = os.path.getmtime( absolute_tag_file )
+        current_mtime = os.path.getmtime( tag_file )
       except:
         continue
-      last_mtime = self.tags_file_last_mtime[ absolute_tag_file ]
+      last_mtime = self._tags_file_last_mtime[ tag_file ]
 
       # We don't want to repeatedly process the same file over and over; we only
       # process if it's changed since the last time we looked at it
       if current_mtime <= last_mtime:
         continue
 
-      self.tags_file_last_mtime[ absolute_tag_file ] = current_mtime
+      self._tags_file_last_mtime[ tag_file ] = current_mtime
-      absolute_paths_to_tag_files.append( absolute_tag_file )
+      absolute_paths_to_tag_files.append( ToUtf8IfNeeded( tag_file ) )
 
     if not absolute_paths_to_tag_files:
       return
 
-    self.completer.AddIdentifiersToDatabaseFromTagFilesAsync(
+    self._completer.AddIdentifiersToDatabaseFromTagFiles(
       absolute_paths_to_tag_files )
 
 
-  def AddIdentifiersFromSyntax( self ):
+  def AddIdentifiersFromSyntax( self, keyword_list, filetypes ):
-    filetype = vim.eval( "&filetype" )
+    keyword_vector = ycm_core.StringVec()
-    if filetype in self.filetypes_with_keywords_loaded:
+    for keyword in keyword_list:
-      return
+      keyword_vector.append( ToUtf8IfNeeded( keyword ) )
 
-    self.filetypes_with_keywords_loaded.add( filetype )
+    filepath = SYNTAX_FILENAME + filetypes[ 0 ]
-
+    self._completer.AddIdentifiersToDatabase( keyword_vector,
-    keyword_set = syntax_parse.SyntaxKeywordsForCurrentBuffer()
+                                             ToUtf8IfNeeded( filetypes[ 0 ] ),
-    keywords = ycm_core.StringVec()
+                                             ToUtf8IfNeeded( filepath ) )
-    for keyword in keyword_set:
-      keywords.append( keyword )
-
-    filepath = SYNTAX_FILENAME + filetype
-    self.completer.AddIdentifiersToDatabase( keywords,
-                                             filetype,
-                                             filepath )
 
 
-  def OnFileReadyToParse( self ):
+  def OnFileReadyToParse( self, request_data ):
-    self.AddBufferIdentifiers()
+    self.AddBufferIdentifiers( request_data )
-
+    if 'tag_files' in request_data:
-    if vimsupport.GetBoolValue( 'g:ycm_collect_identifiers_from_tags_files' ):
+      self.AddIdentifiersFromTagFiles( request_data[ 'tag_files' ] )
-      self.AddIdentifiersFromTagFiles()
+    if 'syntax_keywords' in request_data:
-
+      self.AddIdentifiersFromSyntax( request_data[ 'syntax_keywords' ],
-    if vimsupport.GetBoolValue( 'g:ycm_seed_identifiers_with_syntax' ):
+                                     request_data[ 'filetypes' ] )
-      self.AddIdentifiersFromSyntax()
 
 
-  def OnInsertLeave( self ):
+  def OnInsertLeave( self, request_data ):
-    self.AddIdentifierUnderCursor()
+    self.AddIdentifierUnderCursor( request_data )
 
 
-  def OnCurrentIdentifierFinished( self ):
+  def OnCurrentIdentifierFinished( self, request_data ):
-    self.AddPreviousIdentifier()
+    self.AddPreviousIdentifier( request_data )
 
 
-  def CandidatesFromStoredRequest( self ):
+def _PreviousIdentifier( min_num_completion_start_chars, request_data ):
-    if not self.completions_future:
+  line_num = request_data[ 'line_num' ]
-      return []
+  column_num = request_data[ 'column_num' ]
-    completions = self.completions_future.GetResults()[
+  filepath = request_data[ 'filepath' ]
-      : MAX_IDENTIFIER_COMPLETIONS_RETURNED ]
+  contents_per_line = (
-
+    request_data[ 'file_data' ][ filepath ][ 'contents' ].split( '\n' ) )
-    completions = _RemoveSmallCandidates( completions )
+  line = contents_per_line[ line_num ]
-
-    # We will never have duplicates in completions so with 'dup':1 we tell Vim
-    # to add this candidate even if it's a duplicate of an existing one (which
-    # will never happen). This saves us some expensive string matching
-    # operations in Vim.
-    return [ { 'word': x, 'dup': 1 } for x in completions ]
-
-
-def PreviousIdentifier():
-  line_num, column_num = vimsupport.CurrentLineAndColumn()
-  buffer = vim.current.buffer
-  line = buffer[ line_num ]
 
   end_column = column_num
 
@@ -196,7 +175,7 @@ def PreviousIdentifier():
   # Look at the previous line if we reached the end of the current one
   if end_column == 0:
     try:
-      line = buffer[ line_num - 1]
+      line = contents_per_line[ line_num - 1 ]
     except:
       return ""
     end_column = len( line )
@@ -208,15 +187,52 @@ def PreviousIdentifier():
   while start_column > 0 and utils.IsIdentifierChar( line[ start_column - 1 ] ):
     start_column -= 1
 
-  if end_column - start_column < MIN_NUM_COMPLETION_START_CHARS:
+  if end_column - start_column < min_num_completion_start_chars:
     return ""
 
   return line[ start_column : end_column ]
 
 
-def _RemoveSmallCandidates( candidates ):
+def _RemoveSmallCandidates( candidates, min_num_candidate_size_chars ):
-  if MIN_NUM_CANDIDATE_SIZE_CHARS == 0:
+  if min_num_candidate_size_chars == 0:
     return candidates
 
-  return [ x for x in candidates if len( x ) >= MIN_NUM_CANDIDATE_SIZE_CHARS ]
+  return [ x for x in candidates if len( x ) >= min_num_candidate_size_chars ]
+
+
+# This is meant to behave like 'expand("<cword")' in Vim, thus starting at the
+# cursor column and returning the "cursor word". If the cursor is not on a valid
+# character, it searches forward until a valid identifier is found.
+def _GetCursorIdentifier( request_data ):
+  def FindFirstValidChar( line, column ):
+    current_column = column
+    while not utils.IsIdentifierChar( line[ current_column ] ):
+      current_column += 1
+    return current_column
+
+
+  def FindIdentifierStart( line, valid_char_column ):
+    identifier_start = valid_char_column
+    while identifier_start > 0 and utils.IsIdentifierChar( line[
+      identifier_start - 1 ] ):
+      identifier_start -= 1
+    return identifier_start
+
+
+  def FindIdentifierEnd( line, valid_char_column ):
+    identifier_end = valid_char_column
+    while identifier_end < len( line ) - 1 and utils.IsIdentifierChar( line[
+      identifier_end + 1 ] ):
+      identifier_end += 1
+    return identifier_end + 1
+
+  column_num = request_data[ 'column_num' ]
+  line = request_data[ 'line_value' ]
+
+  try:
+    valid_char_column = FindFirstValidChar( line, column_num )
+    return line[ FindIdentifierStart( line, valid_char_column ) :
+                 FindIdentifierEnd( line, valid_char_column ) ]
+  except:
+    return ''
 

python/ycm/completers/all/omni_completer.py

@@ -19,17 +19,18 @@
 
 import vim
 from ycm import vimsupport
+from ycm import base
 from ycm.completers.completer import Completer
+from ycm.client.base_request import BuildRequestData
 
 OMNIFUNC_RETURNED_BAD_VALUE = 'Omnifunc returned bad value to YCM!'
 OMNIFUNC_NOT_LIST = ( 'Omnifunc did not return a list or a dict with a "words" '
                      ' list when expected.' )
 
 class OmniCompleter( Completer ):
-  def __init__( self ):
+  def __init__( self, user_options ):
-    super( OmniCompleter, self ).__init__()
+    super( OmniCompleter, self ).__init__( user_options )
-    self.omnifunc = None
+    self._omnifunc = None
-    self.stored_candidates = None
 
 
   def SupportedFiletypes( self ):
@@ -37,76 +38,76 @@ class OmniCompleter( Completer ):
 
 
   def ShouldUseCache( self ):
-    return vimsupport.GetBoolValue( "g:ycm_cache_omnifunc" )
+    return bool( self.user_options[ 'cache_omnifunc' ] )
 
 
-  def ShouldUseNow( self, start_column ):
+  # We let the caller call this without passing in request_data. This is useful
-    if self.ShouldUseCache():
+  # for figuring out should we even be preparing the "real" request_data in
-      return super( OmniCompleter, self ).ShouldUseNow( start_column )
+  # omni_completion_request. The real request_data is much bigger and takes
-    return self.ShouldUseNowInner( start_column )
+  # longer to prepare, and we want to avoid creating it twice.
-
+  def ShouldUseNow( self, request_data = None ):
-
+    if not self._omnifunc:
-  def ShouldUseNowInner( self, start_column ):
-    if not self.omnifunc:
       return False
-    return super( OmniCompleter, self ).ShouldUseNowInner( start_column )
 
+    if not request_data:
+      request_data = _BuildRequestDataSubstitute()
 
-  def CandidatesForQueryAsync( self, query, unused_start_column ):
     if self.ShouldUseCache():
-      return super( OmniCompleter, self ).CandidatesForQueryAsync(
+      return super( OmniCompleter, self ).ShouldUseNow( request_data )
-          query, unused_start_column )
+    return self.ShouldUseNowInner( request_data )
+
+
+  def ShouldUseNowInner( self, request_data ):
+    if not self._omnifunc:
+      return False
+    return super( OmniCompleter, self ).ShouldUseNowInner( request_data )
+
+
+  def ComputeCandidates( self, request_data ):
+    if self.ShouldUseCache():
+      return super( OmniCompleter, self ).ComputeCandidates(
+        request_data )
     else:
-      return self.CandidatesForQueryAsyncInner( query, unused_start_column )
+      if self.ShouldUseNowInner( request_data ):
+        return self.ComputeCandidatesInner( request_data )
+      return []
 
 
-  def CandidatesForQueryAsyncInner( self, query, unused_start_column ):
+  def ComputeCandidatesInner( self, request_data ):
-    if not self.omnifunc:
+    if not self._omnifunc:
-      self.stored_candidates = None
+      return []
-      return
 
     try:
-      return_value = int( vim.eval( self.omnifunc + '(1,"")' ) )
+      return_value = int( vim.eval( self._omnifunc + '(1,"")' ) )
       if return_value < 0:
-        self.stored_candidates = None
+        return []
-        return
 
-      omnifunc_call = [ self.omnifunc,
+      omnifunc_call = [ self._omnifunc,
                         "(0,'",
-                        vimsupport.EscapeForVim( query ),
+                        vimsupport.EscapeForVim( request_data[ 'query' ] ),
                         "')" ]
 
       items = vim.eval( ''.join( omnifunc_call ) )
 
       if 'words' in items:
-        items = items['words']
+        items = items[ 'words' ]
       if not hasattr( items, '__iter__' ):
         raise TypeError( OMNIFUNC_NOT_LIST )
 
-      self.stored_candidates = filter( bool, items )
+      return filter( bool, items )
-    except (TypeError, ValueError) as error:
+    except ( TypeError, ValueError ) as error:
       vimsupport.PostVimMessage(
         OMNIFUNC_RETURNED_BAD_VALUE + ' ' + str( error ) )
-      self.stored_candidates = None
+      return []
-      return
 
 
-
+  def OnFileReadyToParse( self, request_data ):
-  def AsyncCandidateRequestReadyInner( self ):
+    self._omnifunc = vim.eval( '&omnifunc' )
-    return True
 
 
-  def OnFileReadyToParse( self ):
+def _BuildRequestDataSubstitute():
-    self.omnifunc = vim.eval( '&omnifunc' )
+  data = BuildRequestData( include_buffer_data = False )
+  data[ 'start_column' ] = base.CompletionStartColumn()
+  return data
 
 
-  def CandidatesFromStoredRequest( self ):
-    if self.ShouldUseCache():
-      return super( OmniCompleter, self ).CandidatesFromStoredRequest()
-    else:
-      return self.CandidatesFromStoredRequestInner()
-
-
-  def CandidatesFromStoredRequestInner( self ):
-    return self.stored_candidates if self.stored_candidates else []
-

python/ycm/completers/all/tests/identifier_completer_test.py

@@ -0,0 +1,127 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+from nose.tools import eq_
+from ycm.completers.all import identifier_completer
+
+
+def GetCursorIdentifier_StartOfLine_test():
+  eq_( 'foo',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 0,
+           'line_value': 'foo'
+         } ) )
+
+  eq_( 'fooBar',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 0,
+           'line_value': 'fooBar'
+         } ) )
+
+
+def GetCursorIdentifier_EndOfLine_test():
+  eq_( 'foo',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 2,
+           'line_value': 'foo'
+         } ) )
+
+
+def GetCursorIdentifier_PastEndOfLine_test():
+  eq_( '',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 10,
+           'line_value': 'foo'
+         } ) )
+
+
+def GetCursorIdentifier_NegativeColumn_test():
+  eq_( '',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': -10,
+           'line_value': 'foo'
+         } ) )
+
+
+def GetCursorIdentifier_StartOfLine_StopsAtNonIdentifierChar_test():
+  eq_( 'foo',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 0,
+           'line_value': 'foo(goo)'
+         } ) )
+
+
+def GetCursorIdentifier_AtNonIdentifier_test():
+  eq_( 'goo',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 3,
+           'line_value': 'foo(goo)'
+         } ) )
+
+
+def GetCursorIdentifier_WalksForwardForIdentifier_test():
+  eq_( 'foo',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 0,
+           'line_value': '       foo'
+         } ) )
+
+
+def GetCursorIdentifier_FindsNothingForward_test():
+  eq_( '',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 4,
+           'line_value': 'foo   ()***()'
+         } ) )
+
+
+def GetCursorIdentifier_SingleCharIdentifier_test():
+  eq_( 'f',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 0,
+           'line_value': '    f    '
+         } ) )
+
+
+def GetCursorIdentifier_StartsInMiddleOfIdentifier_test():
+  eq_( 'foobar',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 3,
+           'line_value': 'foobar'
+         } ) )
+
+
+def GetCursorIdentifier_LineEmpty_test():
+  eq_( '',
+       identifier_completer._GetCursorIdentifier(
+         {
+           'column_num': 11,
+           'line_value': ''
+         } ) )

python/ycm/completers/c/hook.py

@@ -20,9 +20,9 @@
 import ycm_core
 from ycm.completers.cpp.clang_completer import ClangCompleter
 
-def GetCompleter():
+def GetCompleter( user_options ):
   if ycm_core.HasClangSupport():
-    return ClangCompleter()
+    return ClangCompleter( user_options )
   else:
     return None
 

python/ycm/completers/completer.py

@@ -18,16 +18,12 @@
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
 import abc
-import vim
+import ycm_client_support
-import ycm_core
+from ycm.utils import ToUtf8IfNeeded, ForceSemanticCompletion
-from ycm import vimsupport
 from ycm.completers.completer_utils import TriggersForFiletype
 
 NO_USER_COMMANDS = 'This completer does not define any commands.'
 
-MIN_NUM_CHARS = int( vimsupport.GetVariableValue(
-  "g:ycm_min_num_of_chars_for_completion" ) )
-
 class Completer( object ):
   """A base class for all Completers in YCM.
 
@@ -36,10 +32,11 @@ class Completer( object ):
   calling on your Completer:
 
   ShouldUseNow() is called with the start column of where a potential completion
-  string should start. For instance, if the user's input is 'foo.bar' and the
+  string should start and the current line (string) the cursor is on. For
-  cursor is on the 'r' in 'bar', start_column will be the 0-based index of 'b'
+  instance, if the user's input is 'foo.bar' and the cursor is on the 'r' in
-  in the line. Your implementation of ShouldUseNow() should return True if your
+  'bar', start_column will be the 0-based index of 'b' in the line. Your
-  semantic completer should be used and False otherwise.
+  implementation of ShouldUseNow() should return True if your semantic completer
+  should be used and False otherwise.
 
   This is important to get right. You want to return False if you can't provide
   completions because then the identifier completer will kick in, and that's
@@ -61,44 +58,21 @@ class Completer( object ):
   and will NOT re-query your completer but will in fact provide fuzzy-search on
   the candidate strings that were stored in the cache.
 
-  CandidatesForQueryAsync() is the main entry point when the user types. For
+  ComputeCandidates() is the main entry point when the user types. For
   "foo.bar", the user query is "bar" and completions matching this string should
-  be shown. The job of CandidatesForQueryAsync() is to merely initiate this
+  be shown. It should return the list of candidates.  The format of the result
-  request, which will hopefully be processed in a background thread. You may
+  can be a list of strings or a more complicated list of dictionaries. Use
-  want to subclass ThreadedCompleter instead of Completer directly.
+  ycm.server.responses.BuildCompletionData to build the detailed response. See
+  clang_completer.py to see how its used in practice.
 
-  AsyncCandidateRequestReady() is the function that is repeatedly polled until
+  Again, you probably want to override ComputeCandidatesInner().
-  it returns True. If CandidatesForQueryAsync() started a background task of
-  collecting the required completions, AsyncCandidateRequestReady() would check
-  the state of that task and return False until it was completed.
-
-  CandidatesFromStoredRequest() should return the list of candidates. This is
-  what YCM calls after AsyncCandidateRequestReady() returns True. The format of
-  the result can be a list of strings or a more complicated list of
-  dictionaries. See ':h complete-items' for the format, and clang_completer.py
-  to see how its used in practice.
 
   You also need to implement the SupportedFiletypes() function which should
   return a list of strings, where the strings are Vim filetypes your completer
   supports.
 
-  clang_completer.py is a good example of a "complicated" completer that
+  clang_completer.py is a good example of a "complicated" completer. A good
-  maintains its own internal cache and therefore directly overrides the "main"
+  example of a simple completer is ultisnips_completer.py.
-  functions in the API instead of the *Inner versions. A good example of a
-  simple completer that does not do this is omni_completer.py.
-
-  If you're confident your completer doesn't need a background task (think
-  again, you probably do) because you can "certainly" furnish a response in
-  under 10ms, then you can perform your backend processing in a synchronous
-  fashion. You may also need to do this because of technical restrictions (much
-  like omni_completer.py has to do it because accessing Vim internals is not
-  thread-safe). But even if you're certain, still try to do the processing in a
-  background thread. Your completer is unlikely to be merged if it does not,
-  because synchronous processing will block Vim's GUI thread and that's a very,
-  VERY bad thing (so try not to do it!). Again, you may want to subclass
-  ThreadedCompleter instead of Completer directly; ThreadedCompleter will
-  abstract away the use of a background thread for you. See
-  threaded_completer.py.
 
   The On* functions are provided for your convenience. They are called when
   their specific events occur. For instance, the identifier completer collects
@@ -108,42 +82,48 @@ class Completer( object ):
   One special function is OnUserCommand. It is called when the user uses the
   command :YcmCompleter and is passed all extra arguments used on command
   invocation (e.g. OnUserCommand(['first argument', 'second'])).  This can be
-  used for completer-specific commands such as reloading external
+  used for completer-specific commands such as reloading external configuration.
-  configuration.
   When the command is called with no arguments you should print a short summary
   of the supported commands or point the user to the help section where this
-  information can be found."""
+  information can be found.
+
+  Override the Shutdown() member function if your Completer subclass needs to do
+  custom cleanup logic on server shutdown."""
 
   __metaclass__ = abc.ABCMeta
 
-  def __init__( self ):
+  def __init__( self, user_options ):
-    self.triggers_for_filetype = TriggersForFiletype()
+    self.user_options = user_options
+    self.min_num_chars = user_options[ 'min_num_of_chars_for_completion' ]
+    self.triggers_for_filetype = TriggersForFiletype(
+      user_options[ 'semantic_triggers' ] )
     self.completions_future = None
-    self.completions_cache = None
+    self._completions_cache = None
-    self.completion_start_column = None
 
 
   # It's highly likely you DON'T want to override this function but the *Inner
   # version of it.
-  def ShouldUseNow( self, start_column ):
+  def ShouldUseNow( self, request_data ):
-    inner_says_yes = self.ShouldUseNowInner( start_column )
+    inner_says_yes = self.ShouldUseNowInner( request_data )
     if not inner_says_yes:
-      self.completions_cache = None
+      self._completions_cache = None
 
-    previous_results_were_empty = ( self.completions_cache and
+    previous_results_were_empty = ( self._completions_cache and
-                                    self.completions_cache.CacheValid(
+                                    self._completions_cache.CacheValid(
-                                      start_column ) and
+                                      request_data[ 'line_num' ],
-                                    not self.completions_cache.raw_completions )
+                                      request_data[ 'start_column' ] ) and
+                                    not self._completions_cache.raw_completions )
     return inner_says_yes and not previous_results_were_empty
 
 
-  def ShouldUseNowInner( self, start_column ):
+  def ShouldUseNowInner( self, request_data ):
-    line = vim.current.line
+    current_line = request_data[ 'line_value' ]
-    line_length = len( line )
+    start_column = request_data[ 'start_column' ]
+    line_length = len( current_line )
     if not line_length or start_column - 1 >= line_length:
       return False
 
-    filetype = self._CurrentFiletype()
+    filetype = self._CurrentFiletype( request_data[ 'filetypes' ] )
     triggers = self.triggers_for_filetype[ filetype ]
 
     for trigger in triggers:
@@ -151,7 +131,7 @@ class Completer( object ):
       trigger_length = len( trigger )
       while True:
         line_index = start_column + index
-        if line_index < 0 or line[ line_index ] != trigger[ index ]:
+        if line_index < 0 or current_line[ line_index ] != trigger[ index ]:
           break
 
         if abs( index ) == trigger_length:
@@ -160,154 +140,113 @@ class Completer( object ):
     return False
 
 
-  def QueryLengthAboveMinThreshold( self, start_column ):
+  def QueryLengthAboveMinThreshold( self, request_data ):
-    query_length = vimsupport.CurrentColumn() - start_column
+    query_length = request_data[ 'column_num' ] - request_data[ 'start_column' ]
-    return query_length >= MIN_NUM_CHARS
+    return query_length >= self.min_num_chars
+
 
   # It's highly likely you DON'T want to override this function but the *Inner
   # version of it.
-  def CandidatesForQueryAsync( self, query, start_column ):
+  def ComputeCandidates( self, request_data ):
-    self.completion_start_column = start_column
+    if ( not ForceSemanticCompletion( request_data ) and
+         not self.ShouldUseNow( request_data ) ):
+      return []
 
-    if query and self.completions_cache and self.completions_cache.CacheValid(
+    candidates = self._GetCandidatesFromSubclass( request_data )
-      start_column ):
+    if request_data[ 'query' ]:
-      self.completions_cache.filtered_completions = (
+      candidates = self.FilterAndSortCandidates( candidates,
-        self.FilterAndSortCandidates(
+                                                 request_data[ 'query' ] )
-          self.completions_cache.raw_completions,
+    return candidates
-          query ) )
+
+
+  def _GetCandidatesFromSubclass( self, request_data ):
+    if ( self._completions_cache and
+         self._completions_cache.CacheValid( request_data[ 'line_num' ],
+                                             request_data[ 'start_column' ] ) ):
+      return self._completions_cache.raw_completions
     else:
-      self.completions_cache = None
+      self._completions_cache = CompletionsCache()
-      self.CandidatesForQueryAsyncInner( query, start_column )
+      self._completions_cache.raw_completions = self.ComputeCandidatesInner(
+          request_data )
+      self._completions_cache.line = request_data[ 'line_num' ]
+      self._completions_cache.column = request_data[ 'start_column' ]
+      return self._completions_cache.raw_completions
+
+
+  def ComputeCandidatesInner( self, request_data ):
+    pass
 
 
   def DefinedSubcommands( self ):
     return []
 
 
-  def EchoUserCommandsHelpMessage( self ):
+  def UserCommandsHelpMessage( self ):
     subcommands = self.DefinedSubcommands()
     if subcommands:
-      vimsupport.EchoText( 'Supported commands are:\n' +
+      return ( 'Supported commands are:\n' +
-                           '\n'.join( subcommands ) +
+               '\n'.join( subcommands ) +
-                           '\nSee the docs for information on what they do.' )
+               '\nSee the docs for information on what they do.' )
     else:
-      vimsupport.EchoText( 'No supported subcommands' )
+      return 'This Completer has no supported subcommands.'
 
 
   def FilterAndSortCandidates( self, candidates, query ):
     if not candidates:
       return []
 
-    if hasattr( candidates, 'words' ):
+    # We need to handle both an omni_completer style completer and a server
-      candidates = candidates.words
+    # style completer
-    items_are_objects = 'word' in candidates[ 0 ]
+    if 'words' in candidates:
+      candidates = candidates[ 'words' ]
 
-    matches = ycm_core.FilterAndSortCandidates(
+    sort_property = ''
+    if 'word' in candidates[ 0 ]:
+      sort_property = 'word'
+    elif 'insertion_text' in candidates[ 0 ]:
+      sort_property = 'insertion_text'
+
+    matches = ycm_client_support.FilterAndSortCandidates(
       candidates,
-      'word' if items_are_objects else '',
+      sort_property,
-      query )
+      ToUtf8IfNeeded( query ) )
 
     return matches
 
 
-  def CandidatesForQueryAsyncInner( self, query, start_column ):
+  def OnFileReadyToParse( self, request_data ):
     pass
 
 
-  # It's highly likely you DON'T want to override this function but the *Inner
+  def OnBufferVisit( self, request_data ):
-  # version of it.
-  def AsyncCandidateRequestReady( self ):
-    if self.completions_cache:
-      return True
-    else:
-      return self.AsyncCandidateRequestReadyInner()
-
-
-  def AsyncCandidateRequestReadyInner( self ):
-    if not self.completions_future:
-      # We return True so that the caller can extract the default value from the
-      # future
-      return True
-    return self.completions_future.ResultsReady()
-
-
-  # It's highly likely you DON'T want to override this function but the *Inner
-  # version of it.
-  def CandidatesFromStoredRequest( self ):
-    if self.completions_cache:
-      return self.completions_cache.filtered_completions
-    else:
-      self.completions_cache = CompletionsCache()
-      self.completions_cache.raw_completions = self.CandidatesFromStoredRequestInner()
-      self.completions_cache.line, _ = vimsupport.CurrentLineAndColumn()
-      self.completions_cache.column = self.completion_start_column
-      return self.completions_cache.raw_completions
-
-
-  def CandidatesFromStoredRequestInner( self ):
-    if not self.completions_future:
-      return []
-    return self.completions_future.GetResults()
-
-
-  def OnFileReadyToParse( self ):
     pass
 
 
-  def OnCursorMovedInsertMode( self ):
+  def OnBufferUnload( self, request_data ):
     pass
 
 
-  def OnCursorMovedNormalMode( self ):
+  def OnInsertLeave( self, request_data ):
     pass
 
 
-  def OnBufferVisit( self ):
+  def OnUserCommand( self, arguments, request_data ):
+    raise NotImplementedError( NO_USER_COMMANDS )
+
+
+  def OnCurrentIdentifierFinished( self, request_data ):
     pass
 
 
-  def OnBufferUnload( self, deleted_buffer_file ):
+  def GetDiagnosticsForCurrentFile( self, request_data ):
-    pass
-
-
-  def OnCursorHold( self ):
-    pass
-
-
-  def OnInsertLeave( self ):
-    pass
-
-
-  def OnVimLeave( self ):
-    pass
-
-
-  def OnUserCommand( self, arguments ):
-    vimsupport.PostVimMessage( NO_USER_COMMANDS )
-
-
-  def OnCurrentIdentifierFinished( self ):
-    pass
-
-
-  def DiagnosticsForCurrentFileReady( self ):
-    return False
-
-
-  def GetDiagnosticsForCurrentFile( self ):
     return []
 
 
-  def ShowDetailedDiagnostic( self ):
+  def GetDetailedDiagnostic( self, request_data ):
     pass
 
 
-  def GettingCompletions( self ):
+  def _CurrentFiletype( self, filetypes ):
-    return False
-
-
-  def _CurrentFiletype( self ):
-    filetypes = vimsupport.CurrentFiletypes()
     supported = self.SupportedFiletypes()
 
     for filetype in filetypes:
@@ -322,10 +261,14 @@ class Completer( object ):
     pass
 
 
-  def DebugInfo( self ):
+  def DebugInfo( self, request_data ):
     return ''
 
 
+  def Shutdown( self ):
+    pass
+
+
 class CompletionsCache( object ):
   def __init__( self ):
     self.line = -1
@@ -334,9 +277,7 @@ class CompletionsCache( object ):
     self.filtered_completions = []
 
 
-  def CacheValid( self, start_column ):
+  def CacheValid( self, current_line, start_column ):
-    completion_line, _ = vimsupport.CurrentLineAndColumn()
+    return current_line == self.line and start_column == self.column
-    completion_column = start_column
-    return completion_line == self.line and completion_column == self.column
 
 

python/ycm/completers/completer_utils.py

@@ -19,7 +19,7 @@
 
 from collections import defaultdict
 from copy import deepcopy
-import vim
+import os
 
 DEFAULT_FILETYPE_TRIGGERS = {
   'c' : ['->', '.'],
@@ -58,12 +58,21 @@ def _FiletypeDictUnion( dict_one, dict_two ):
   return final_dict
 
 
-def TriggersForFiletype():
+def TriggersForFiletype( user_triggers ):
-  user_triggers = _FiletypeTriggerDictFromSpec(
-    vim.eval( 'g:ycm_semantic_triggers' ) )
-
   default_triggers = _FiletypeTriggerDictFromSpec(
     DEFAULT_FILETYPE_TRIGGERS )
 
-  return _FiletypeDictUnion( default_triggers, user_triggers )
+  return _FiletypeDictUnion( default_triggers, dict( user_triggers ) )
 
+
+def _PathToCompletersFolder():
+  dir_of_current_script = os.path.dirname( os.path.abspath( __file__ ) )
+  return os.path.join( dir_of_current_script )
+
+
+def PathToFiletypeCompleterPluginLoader( filetype ):
+  return os.path.join( _PathToCompletersFolder(), filetype, 'hook.py' )
+
+
+def FiletypeCompleterExistsForFiletype( filetype ):
+  return os.path.exists( PathToFiletypeCompleterPluginLoader( filetype ) )

python/ycm/completers/cpp/clang_completer.py

@@ -18,116 +18,84 @@
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
 from collections import defaultdict
-import vim
 import ycm_core
-from ycm import vimsupport
+from ycm.server import responses
 from ycm import extra_conf_store
+from ycm.utils import ToUtf8IfNeeded
 from ycm.completers.completer import Completer
-from ycm.completers.cpp.flags import Flags
+from ycm.completers.cpp.flags import Flags, PrepareFlagsForClang
 
 CLANG_FILETYPES = set( [ 'c', 'cpp', 'objc', 'objcpp' ] )
-MAX_DIAGNOSTICS_TO_DISPLAY = int( vimsupport.GetVariableValue(
+MIN_LINES_IN_FILE_TO_PARSE = 5
-  "g:ycm_max_diagnostics_to_display" ) )
+PARSING_FILE_MESSAGE = 'Still parsing file, no completions yet.'
+NO_COMPILE_FLAGS_MESSAGE = 'Still no compile flags, no completions yet.'
+INVALID_FILE_MESSAGE = 'File is invalid.'
+NO_COMPLETIONS_MESSAGE = 'No completions found; errors in the file?'
+FILE_TOO_SHORT_MESSAGE = (
+  'File is less than {0} lines long; not compiling.'.format(
+    MIN_LINES_IN_FILE_TO_PARSE ) )
+NO_DIAGNOSTIC_MESSAGE = 'No diagnostic for current line!'
 
 
 class ClangCompleter( Completer ):
-  def __init__( self ):
+  def __init__( self, user_options ):
-    super( ClangCompleter, self ).__init__()
+    super( ClangCompleter, self ).__init__( user_options )
-    self.completer = ycm_core.ClangCompleter()
+    self._max_diagnostics_to_display = user_options[
-    self.completer.EnableThreading()
+      'max_diagnostics_to_display' ]
-    self.contents_holder = []
+    self._completer = ycm_core.ClangCompleter()
-    self.filename_holder = []
+    self._flags = Flags()
-    self.last_prepared_diagnostics = []
+    self._diagnostic_store = None
-    self.parse_future = None
-    self.flags = Flags()
-    self.diagnostic_store = None
-
-    # We set this flag when a compilation request comes in while one is already
-    # in progress. We use this to trigger the pending request after the previous
-    # one completes (from GetDiagnosticsForCurrentFile because that's the only
-    # method that knows when the compilation has finished).
-    self.extra_parse_desired = False
 
 
   def SupportedFiletypes( self ):
     return CLANG_FILETYPES
 
 
-  def GetUnsavedFilesVector( self ):
+  def GetUnsavedFilesVector( self, request_data ):
-    # CAREFUL HERE! For UnsavedFile filename and contents we are referring
-    # directly to Python-allocated and -managed memory since we are accepting
-    # pointers to data members of python objects. We need to ensure that those
-    # objects outlive our UnsavedFile objects. This is why we need the
-    # contents_holder and filename_holder lists, to make sure the string objects
-    # are still around when we call CandidatesForQueryAndLocationInFile.  We do
-    # this to avoid an extra copy of the entire file contents.
-
     files = ycm_core.UnsavedFileVec()
-    self.contents_holder = []
+    for filename, file_data in request_data[ 'file_data' ].iteritems():
-    self.filename_holder = []
+      if not ClangAvailableForFiletypes( file_data[ 'filetypes' ] ):
-    for buffer in vimsupport.GetUnsavedBuffers():
-      if not ClangAvailableForBuffer( buffer ):
         continue
-      contents = '\n'.join( buffer )
+      contents = file_data[ 'contents' ]
-      name = buffer.name
+      if not contents or not filename:
-      if not contents or not name:
         continue
-      self.contents_holder.append( contents )
-      self.filename_holder.append( name )
 
       unsaved_file = ycm_core.UnsavedFile()
-      unsaved_file.contents_ = self.contents_holder[ -1 ]
+      utf8_contents = ToUtf8IfNeeded( contents )
-      unsaved_file.length_ = len( self.contents_holder[ -1 ] )
+      unsaved_file.contents_ = utf8_contents
-      unsaved_file.filename_ = self.filename_holder[ -1 ]
+      unsaved_file.length_ = len( utf8_contents )
+      unsaved_file.filename_ = ToUtf8IfNeeded( filename )
 
       files.append( unsaved_file )
-
     return files
 
 
-  def CandidatesForQueryAsync( self, query, start_column ):
+  def ComputeCandidatesInner( self, request_data ):
-    filename = vim.current.buffer.name
+    filename = request_data[ 'filepath' ]
-
     if not filename:
       return
 
-    if self.completer.UpdatingTranslationUnit( filename ):
+    if self._completer.UpdatingTranslationUnit( ToUtf8IfNeeded( filename ) ):
-      vimsupport.PostVimMessage( 'Still parsing file, no completions yet.' )
+      raise RuntimeError( PARSING_FILE_MESSAGE )
-      self.completions_future = None
-      return
 
-    flags = self.flags.FlagsForFile( filename )
+    flags = self._FlagsForRequest( request_data )
     if not flags:
-      vimsupport.PostVimMessage( 'Still no compile flags, no completions yet.' )
+      raise RuntimeError( NO_COMPILE_FLAGS_MESSAGE )
-      self.completions_future = None
-      return
 
-    # TODO: sanitize query, probably in C++ code
+    files = self.GetUnsavedFilesVector( request_data )
-
+    line = request_data[ 'line_num' ] + 1
-    files = ycm_core.UnsavedFileVec()
+    column = request_data[ 'start_column' ] + 1
-    if not query:
+    results = self._completer.CandidatesForLocationInFile(
-      files = self.GetUnsavedFilesVector()
+        ToUtf8IfNeeded( filename ),
-
-    line, _ = vim.current.window.cursor
-    column = start_column + 1
-    self.completions_future = (
-      self.completer.CandidatesForQueryAndLocationInFileAsync(
-        query,
-        filename,
         line,
         column,
         files,
-        flags ) )
+        flags )
 
-
-  def CandidatesFromStoredRequest( self ):
-    if not self.completions_future:
-      return []
-    results = [ CompletionDataToDict( x ) for x in
-                self.completions_future.GetResults() ]
     if not results:
-      vimsupport.PostVimMessage( 'No completions found; errors in the file?' )
+      raise RuntimeError( NO_COMPLETIONS_MESSAGE )
-    return results
+
+    return [ ConvertCompletionData( x ) for x in results ]
 
 
   def DefinedSubcommands( self ):
@@ -137,157 +105,118 @@ class ClangCompleter( Completer ):
              'ClearCompilationFlagCache']
 
 
-  def OnUserCommand( self, arguments ):
+  def OnUserCommand( self, arguments, request_data ):
     if not arguments:
-      self.EchoUserCommandsHelpMessage()
+      raise ValueError( self.UserCommandsHelpMessage() )
-      return
 
     command = arguments[ 0 ]
     if command == 'GoToDefinition':
-      self._GoToDefinition()
+      return self._GoToDefinition( request_data )
     elif command == 'GoToDeclaration':
-      self._GoToDeclaration()
+      return self._GoToDeclaration( request_data )
     elif command == 'GoToDefinitionElseDeclaration':
-      self._GoToDefinitionElseDeclaration()
+      return self._GoToDefinitionElseDeclaration( request_data )
     elif command == 'ClearCompilationFlagCache':
-      self._ClearCompilationFlagCache()
+      return self._ClearCompilationFlagCache()
+    raise ValueError( self.UserCommandsHelpMessage() )
 
 
-  def _LocationForGoTo( self, goto_function ):
+  def _LocationForGoTo( self, goto_function, request_data ):
-    filename = vim.current.buffer.name
+    filename = request_data[ 'filepath' ]
     if not filename:
-      return None
+      raise ValueError( INVALID_FILE_MESSAGE )
 
-    flags = self.flags.FlagsForFile( filename )
+    flags = self._FlagsForRequest( request_data )
     if not flags:
-      vimsupport.PostVimMessage( 'Still no compile flags, can\'t compile.' )
+      raise ValueError( NO_COMPILE_FLAGS_MESSAGE )
-      return None
 
-    files = self.GetUnsavedFilesVector()
+    files = self.GetUnsavedFilesVector( request_data )
-    line, column = vimsupport.CurrentLineAndColumn()
+    line = request_data[ 'line_num' ] + 1
-    # Making the line & column 1-based instead of 0-based
+    column = request_data[ 'column_num' ] + 1
-    line += 1
+    return getattr( self._completer, goto_function )(
-    column += 1
+        ToUtf8IfNeeded( filename ),
-    return getattr( self.completer, goto_function )(
-        filename,
         line,
         column,
         files,
         flags )
 
 
-  def _GoToDefinition( self ):
+  def _GoToDefinition( self, request_data ):
-    location = self._LocationForGoTo( 'GetDefinitionLocation' )
+    location = self._LocationForGoTo( 'GetDefinitionLocation', request_data )
     if not location or not location.IsValid():
-      vimsupport.PostVimMessage( 'Can\'t jump to definition.' )
+      raise RuntimeError( 'Can\'t jump to definition.' )
-      return
 
-    vimsupport.JumpToLocation( location.filename_,
+    return responses.BuildGoToResponse( location.filename_,
-                               location.line_number_,
+                                        location.line_number_ - 1,
-                               location.column_number_ )
+                                        location.column_number_ - 1)
 
 
-  def _GoToDeclaration( self ):
+  def _GoToDeclaration( self, request_data ):
-    location = self._LocationForGoTo( 'GetDeclarationLocation' )
+    location = self._LocationForGoTo( 'GetDeclarationLocation', request_data )
     if not location or not location.IsValid():
-      vimsupport.PostVimMessage( 'Can\'t jump to declaration.' )
+      raise RuntimeError( 'Can\'t jump to declaration.' )
-      return
 
-    vimsupport.JumpToLocation( location.filename_,
+    return responses.BuildGoToResponse( location.filename_,
-                               location.line_number_,
+                                        location.line_number_ - 1,
-                               location.column_number_ )
+                                        location.column_number_ - 1)
 
 
-  def _GoToDefinitionElseDeclaration( self ):
+  def _GoToDefinitionElseDeclaration( self, request_data ):
-    location = self._LocationForGoTo( 'GetDefinitionLocation' )
+    location = self._LocationForGoTo( 'GetDefinitionLocation', request_data )
     if not location or not location.IsValid():
-      location = self._LocationForGoTo( 'GetDeclarationLocation' )
+      location = self._LocationForGoTo( 'GetDeclarationLocation', request_data )
     if not location or not location.IsValid():
-      vimsupport.PostVimMessage( 'Can\'t jump to definition or declaration.' )
+      raise RuntimeError( 'Can\'t jump to definition or declaration.' )
-      return
+
+    return responses.BuildGoToResponse( location.filename_,
+                                        location.line_number_ - 1,
+                                        location.column_number_ - 1)
 
-    vimsupport.JumpToLocation( location.filename_,
-                               location.line_number_,
-                               location.column_number_ )
 
 
   def _ClearCompilationFlagCache( self ):
-    self.flags.Clear()
+    self._flags.Clear()
 
 
+  def OnFileReadyToParse( self, request_data ):
+    filename = request_data[ 'filepath' ]
+    contents = request_data[ 'file_data' ][ filename ][ 'contents' ]
+    if contents.count( '\n' ) < MIN_LINES_IN_FILE_TO_PARSE:
+      raise ValueError( FILE_TOO_SHORT_MESSAGE )
 
-  def OnFileReadyToParse( self ):
-    if vimsupport.NumLinesInBuffer( vim.current.buffer ) < 5:
-      self.parse_future = None
-      return
-
-    filename = vim.current.buffer.name
     if not filename:
-      return
+      raise ValueError( INVALID_FILE_MESSAGE )
 
-    if self.completer.UpdatingTranslationUnit( filename ):
+    flags = self._FlagsForRequest( request_data )
-      self.extra_parse_desired = True
-      return
-
-    flags = self.flags.FlagsForFile( filename )
     if not flags:
-      self.parse_future = None
+      raise ValueError( NO_COMPILE_FLAGS_MESSAGE )
-      return
 
-    self.parse_future = self.completer.UpdateTranslationUnitAsync(
+    diagnostics = self._completer.UpdateTranslationUnit(
-      filename,
+      ToUtf8IfNeeded( filename ),
-      self.GetUnsavedFilesVector(),
+      self.GetUnsavedFilesVector( request_data ),
       flags )
 
-    self.extra_parse_desired = False
+    self._diagnostic_store = DiagnosticsToDiagStructure( diagnostics )
+    return [ ConvertToDiagnosticResponse( x ) for x in
+             diagnostics[ : self._max_diagnostics_to_display ] ]
 
 
-  def OnBufferUnload( self, deleted_buffer_file ):
+  def OnBufferUnload( self, request_data ):
-    self.completer.DeleteCachesForFileAsync( deleted_buffer_file )
+    self._completer.DeleteCachesForFile(
+        ToUtf8IfNeeded( request_data[ 'unloaded_buffer' ] ) )
 
 
-  def DiagnosticsForCurrentFileReady( self ):
+  def GetDetailedDiagnostic( self, request_data ):
-    if not self.parse_future:
+    current_line = request_data[ 'line_num' ] + 1
-      return False
+    current_column = request_data[ 'column_num' ] + 1
+    current_file = request_data[ 'filepath' ]
 
-    return self.parse_future.ResultsReady()
+    if not self._diagnostic_store:
+      raise ValueError( NO_DIAGNOSTIC_MESSAGE )
 
-
+    diagnostics = self._diagnostic_store[ current_file ][ current_line ]
-  def GettingCompletions( self ):
-    return self.completer.UpdatingTranslationUnit( vim.current.buffer.name )
-
-
-  def GetDiagnosticsForCurrentFile( self ):
-    if self.DiagnosticsForCurrentFileReady():
-      diagnostics = self.completer.DiagnosticsForFile( vim.current.buffer.name )
-      self.diagnostic_store = DiagnosticsToDiagStructure( diagnostics )
-      self.last_prepared_diagnostics = [ DiagnosticToDict( x ) for x in
-          diagnostics[ : MAX_DIAGNOSTICS_TO_DISPLAY ] ]
-      self.parse_future = None
-
-      if self.extra_parse_desired:
-        self.OnFileReadyToParse()
-
-    return self.last_prepared_diagnostics
-
-
-  def ShowDetailedDiagnostic( self ):
-    current_line, current_column = vimsupport.CurrentLineAndColumn()
-
-    # CurrentLineAndColumn() numbers are 0-based, clang numbers are 1-based
-    current_line += 1
-    current_column += 1
-
-    current_file = vim.current.buffer.name
-
-    if not self.diagnostic_store:
-      vimsupport.PostVimMessage( "No diagnostic for current line!" )
-      return
-
-    diagnostics = self.diagnostic_store[ current_file ][ current_line ]
     if not diagnostics:
-      vimsupport.PostVimMessage( "No diagnostic for current line!" )
+      raise ValueError( NO_DIAGNOSTIC_MESSAGE )
-      return
 
     closest_diagnostic = None
     distance_to_closest_diagnostic = 999
@@ -298,50 +227,36 @@ class ClangCompleter( Completer ):
         distance_to_closest_diagnostic = distance
         closest_diagnostic = diagnostic
 
-    vimsupport.EchoText( closest_diagnostic.long_formatted_text_ )
+    return responses.BuildDisplayMessageResponse(
+      closest_diagnostic.long_formatted_text_ )
 
 
-  def ShouldUseNow( self, start_column ):
+  def DebugInfo( self, request_data ):
-    # We don't want to use the Completer API cache, we use one in the C++ code.
+    filename = request_data[ 'filepath' ]
-    return self.ShouldUseNowInner( start_column )
-
-
-  def DebugInfo( self ):
-    filename = vim.current.buffer.name
     if not filename:
       return ''
-    flags = self.flags.FlagsForFile( filename ) or []
+    flags = self._FlagsForRequest( request_data ) or []
     source = extra_conf_store.ModuleFileForSourceFile( filename )
     return 'Flags for {0} loaded from {1}:\n{2}'.format( filename,
                                                          source,
                                                          list( flags ) )
 
 
-# TODO: make these functions module-local
+  def _FlagsForRequest( self, request_data ):
-def CompletionDataToDict( completion_data ):
+    filename = request_data[ 'filepath' ]
-  # see :h complete-items for a description of the dictionary fields
+    if 'compilation_flags' in request_data:
-  return {
+      return PrepareFlagsForClang( request_data[ 'compilation_flags' ],
-    'word' : completion_data.TextToInsertInBuffer(),
+                                   filename )
-    'abbr' : completion_data.MainCompletionText(),
+    return self._flags.FlagsForFile( filename )
-    'menu' : completion_data.ExtraMenuInfo(),
-    'kind' : completion_data.kind_,
-    'info' : completion_data.DetailedInfoForPreviewWindow(),
-    'dup'  : 1,
-  }
 
 
-def DiagnosticToDict( diagnostic ):
+def ConvertCompletionData( completion_data ):
-  # see :h getqflist for a description of the dictionary fields
+  return responses.BuildCompletionData(
-  return {
+    insertion_text = completion_data.TextToInsertInBuffer(),
-    # TODO: wrap the bufnr generation into a function
+    menu_text = completion_data.MainCompletionText(),
-    'bufnr' : int( vim.eval( "bufnr('{0}', 1)".format(
+    extra_menu_info = completion_data.ExtraMenuInfo(),
-      diagnostic.filename_ ) ) ),
+    kind = completion_data.kind_,
-    'lnum'  : diagnostic.line_number_,
+    detailed_info = completion_data.DetailedInfoForPreviewWindow() )
-    'col'   : diagnostic.column_number_,
-    'text'  : diagnostic.text_,
-    'type'  : diagnostic.kind_,
-    'valid' : 1
-  }
 
 
 def DiagnosticsToDiagStructure( diagnostics ):
@@ -352,12 +267,19 @@ def DiagnosticsToDiagStructure( diagnostics ):
   return structure
 
 
-def ClangAvailableForBuffer( buffer_object ):
+def ClangAvailableForFiletypes( filetypes ):
-  filetypes = vimsupport.FiletypesForBuffer( buffer_object )
   return any( [ filetype in CLANG_FILETYPES for filetype in filetypes ] )
 
 
-def InCFamilyFile():
+def InCFamilyFile( filetypes ):
-  return any( [ filetype in CLANG_FILETYPES for filetype in
+  return ClangAvailableForFiletypes( filetypes )
-                vimsupport.CurrentFiletypes() ] )
+
+
+def ConvertToDiagnosticResponse( diagnostic ):
+  return responses.BuildDiagnosticData( diagnostic.filename_,
+                                        diagnostic.line_number_ - 1,
+                                        diagnostic.column_number_ - 1,
+                                        diagnostic.text_,
+                                        diagnostic.kind_ )
+
 

python/ycm/completers/cpp/flags.py

@@ -19,12 +19,12 @@
 
 import ycm_core
 import os
-from ycm import vimsupport
 from ycm import extra_conf_store
+from ycm.utils import ToUtf8IfNeeded
 
-NO_EXTRA_CONF_FILENAME_MESSAGE = ('No {0} file detected, so no compile flags '
+NO_EXTRA_CONF_FILENAME_MESSAGE = ( 'No {0} file detected, so no compile flags '
   'are available. Thus no semantic support for C/C++/ObjC/ObjC++. Go READ THE '
-  'DOCS *NOW*, DON\'T file a bug report.').format(
+  'DOCS *NOW*, DON\'T file a bug report.' ).format(
     extra_conf_store.YCM_EXTRA_CONF_FILENAME )
 
 
@@ -47,8 +47,8 @@ class Flags( object ):
       module = extra_conf_store.ModuleForSourceFile( filename )
       if not module:
         if not self.no_extra_conf_file_warning_posted:
-          vimsupport.PostVimMessage( NO_EXTRA_CONF_FILENAME_MESSAGE )
           self.no_extra_conf_file_warning_posted = True
+          raise RuntimeError( NO_EXTRA_CONF_FILENAME_MESSAGE )
         return None
 
       results = module.FlagsForFile( filename )
@@ -59,7 +59,7 @@ class Flags( object ):
       flags = list( results[ 'flags' ] )
       if add_special_clang_flags:
         flags += self.special_clang_flags
-      sanitized_flags = _PrepareFlagsForClang( flags, filename )
+      sanitized_flags = PrepareFlagsForClang( flags, filename )
 
       if results[ 'do_cache' ]:
         self.flags_for_file[ filename ] = sanitized_flags
@@ -95,7 +95,7 @@ class Flags( object ):
     self.flags_for_file.clear()
 
 
-def _PrepareFlagsForClang( flags, filename ):
+def PrepareFlagsForClang( flags, filename ):
   flags = _RemoveUnusedFlags( flags, filename )
   flags = _SanitizeFlags( flags )
   return flags
@@ -121,7 +121,7 @@ def _SanitizeFlags( flags ):
 
   vector = ycm_core.StringVec()
   for flag in sanitized_flags:
-    vector.append( flag )
+    vector.append( ToUtf8IfNeeded( flag ) )
   return vector
 
 

python/ycm/completers/cpp/hook.py

@@ -20,9 +20,9 @@
 import ycm_core
 from ycm.completers.cpp.clang_completer import ClangCompleter
 
-def GetCompleter():
+def GetCompleter( user_options ):
   if ycm_core.HasClangSupport():
-    return ClangCompleter()
+    return ClangCompleter( user_options )
   else:
     return None
 

python/ycm/completers/cs/cs_completer.py

@@ -18,39 +18,37 @@
 # You should have received a copy of the GNU General Public License
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
-import vim
 import os
 from sys import platform
 import glob
-from ycm.completers.threaded_completer import ThreadedCompleter
+from ycm.completers.completer import Completer
-from ycm import vimsupport
+from ycm.server import responses
+from ycm import utils
 import urllib2
 import urllib
 import urlparse
 import json
 import subprocess
-import tempfile
+import logging
-
 
 SERVER_NOT_FOUND_MSG = ( 'OmniSharp server binary not found at {0}. ' +
 'Did you compile it? You can do so by running ' +
 '"./install.sh --omnisharp-completer".' )
 
-class CsharpCompleter( ThreadedCompleter ):
+
+class CsharpCompleter( Completer ):
   """
   A Completer that uses the Omnisharp server as completion engine.
   """
 
-  def __init__( self ):
+  def __init__( self, user_options ):
-    super( CsharpCompleter, self ).__init__()
+    super( CsharpCompleter, self ).__init__( user_options )
     self._omnisharp_port = None
-
+    self._logger = logging.getLogger( __name__ )
-    if vimsupport.GetBoolValue( 'g:ycm_auto_start_csharp_server' ):
-      self._StartServer()
 
 
-  def OnVimLeave( self ):
+  def Shutdown( self ):
-    if ( vimsupport.GetBoolValue( 'g:ycm_auto_stop_csharp_server' ) and
+    if ( self.user_options[ 'auto_start_csharp_server' ] and
          self._ServerIsRunning() ):
       self._StopServer()
 
@@ -60,79 +58,83 @@ class CsharpCompleter( ThreadedCompleter ):
     return [ 'cs' ]
 
 
-  def ComputeCandidates( self, unused_query, unused_start_column ):
+  def ComputeCandidatesInner( self, request_data ):
-    return [ { 'word': str( completion[ 'CompletionText' ] ),
+    return [ responses.BuildCompletionData(
-               'menu': str( completion[ 'DisplayText' ] ),
+                completion[ 'CompletionText' ],
-               'info': str( completion[ 'Description' ] ) }
+                completion[ 'DisplayText' ],
-             for completion in self._GetCompletions() ]
+                completion[ 'Description' ] )
+             for completion in self._GetCompletions( request_data ) ]
 
 
   def DefinedSubcommands( self ):
     return [ 'StartServer',
              'StopServer',
              'RestartServer',
+             'ServerRunning',
              'GoToDefinition',
              'GoToDeclaration',
              'GoToDefinitionElseDeclaration' ]
 
 
-  def OnUserCommand( self, arguments ):
+  def OnFileReadyToParse( self, request_data ):
+    if ( not self._omnisharp_port and
+         self.user_options[ 'auto_start_csharp_server' ] ):
+      self._StartServer( request_data )
+
+
+  def OnUserCommand( self, arguments, request_data ):
     if not arguments:
-      self.EchoUserCommandsHelpMessage()
+      raise ValueError( self.UserCommandsHelpMessage() )
-      return
 
     command = arguments[ 0 ]
     if command == 'StartServer':
-      self._StartServer()
+      self._StartServer( request_data )
     elif command == 'StopServer':
       self._StopServer()
     elif command == 'RestartServer':
       if self._ServerIsRunning():
         self._StopServer()
-      self._StartServer()
+      self._StartServer( request_data )
+    elif command == 'ServerRunning':
+      return self._ServerIsRunning()
     elif command in [ 'GoToDefinition',
                       'GoToDeclaration',
                       'GoToDefinitionElseDeclaration' ]:
-      self._GoToDefinition()
+      return self._GoToDefinition( request_data )
+    raise ValueError( self.UserCommandsHelpMessage() )
 
 
   def DebugInfo( self ):
     if self._ServerIsRunning():
-      return 'Server running at: {}\nLogfiles:\n{}\n{}'.format(
+      return 'Server running at: {0}\nLogfiles:\n{1}\n{2}'.format(
-        self._PortToHost(), self._filename_stdout, self._filename_stderr )
+        self._ServerLocation(), self._filename_stdout, self._filename_stderr )
     else:
       return 'Server is not running'
 
 
-  def _StartServer( self ):
+  def _StartServer( self, request_data ):
     """ Start the OmniSharp server """
-    self._omnisharp_port = self._FindFreePort()
+    self._logger.info( 'startup' )
-    solutionfiles, folder = _FindSolutionFiles()
+
+    self._omnisharp_port = utils.GetUnusedLocalhostPort()
+    solutionfiles, folder = _FindSolutionFiles( request_data[ 'filepath' ] )
 
     if len( solutionfiles ) == 0:
-      vimsupport.PostVimMessage(
+      raise RuntimeError(
-              'Error starting OmniSharp server: no solutionfile found' )
+        'Error starting OmniSharp server: no solutionfile found' )
-      return
     elif len( solutionfiles ) == 1:
       solutionfile = solutionfiles[ 0 ]
     else:
-      choice = vimsupport.PresentDialog(
+      raise RuntimeError(
-              'Which solutionfile should be loaded?',
+        'Found multiple solution files instead of one!\n{0}'.format(
-              [ str( i ) + " " + solution for i, solution in
+          solutionfiles ) )
-                enumerate( solutionfiles ) ] )
-      if choice == -1:
-        vimsupport.PostVimMessage( 'OmniSharp not started' )
-        return
-      else:
-        solutionfile = solutionfiles[ choice ]
 
     omnisharp = os.path.join(
       os.path.abspath( os.path.dirname( __file__ ) ),
       'OmniSharpServer/OmniSharp/bin/Debug/OmniSharp.exe' )
 
     if not os.path.isfile( omnisharp ):
-      vimsupport.PostVimMessage( SERVER_NOT_FOUND_MSG.format( omnisharp ) )
+      raise RuntimeError( SERVER_NOT_FOUND_MSG.format( omnisharp ) )
-      return
 
     if not platform.startswith( 'win' ):
       omnisharp = 'mono ' + omnisharp
@@ -142,8 +144,8 @@ class CsharpCompleter( ThreadedCompleter ):
     command = [ omnisharp + ' -p ' + str( self._omnisharp_port ) + ' -s ' +
                 path_to_solutionfile ]
 
-    filename_format = ( tempfile.gettempdir() +
+    filename_format = os.path.join( utils.PathToTempDir(),
-                        '/omnisharp_{port}_{sln}_{std}.log' )
+                                   'omnisharp_{port}_{sln}_{std}.log' )
 
     self._filename_stdout = filename_format.format(
         port=self._omnisharp_port, sln=solutionfile, std='stdout' )
@@ -154,82 +156,72 @@ class CsharpCompleter( ThreadedCompleter ):
       with open( self._filename_stdout, 'w' ) as fstdout:
         subprocess.Popen( command, stdout=fstdout, stderr=fstderr, shell=True )
 
-    vimsupport.PostVimMessage( 'Starting OmniSharp server' )
+    self._logger.info( 'Starting OmniSharp server' )
 
 
   def _StopServer( self ):
     """ Stop the OmniSharp server """
     self._GetResponse( '/stopserver' )
     self._omnisharp_port = None
-    vimsupport.PostVimMessage( 'Stopping OmniSharp server' )
+    self._logger.info( 'Stopping OmniSharp server' )
 
 
-  def _GetCompletions( self ):
+  def _GetCompletions( self, request_data ):
     """ Ask server for completions """
-    completions = self._GetResponse( '/autocomplete', self._DefaultParameters() )
+    completions = self._GetResponse( '/autocomplete',
+                                     self._DefaultParameters( request_data ) )
     return completions if completions != None else []
 
 
-  def _GoToDefinition( self ):
+  def _GoToDefinition( self, request_data ):
     """ Jump to definition of identifier under cursor """
-    definition = self._GetResponse( '/gotodefinition', self._DefaultParameters() )
+    definition = self._GetResponse( '/gotodefinition',
+                                    self._DefaultParameters( request_data ) )
     if definition[ 'FileName' ] != None:
-      vimsupport.JumpToLocation( definition[ 'FileName' ],
+      return responses.BuildGoToResponse( definition[ 'FileName' ],
-                                 definition[ 'Line' ],
+                                          definition[ 'Line' ],
-                                 definition[ 'Column' ] )
+                                          definition[ 'Column' ] )
     else:
-      vimsupport.PostVimMessage( 'Can\'t jump to definition' )
+      raise RuntimeError( 'Can\'t jump to definition' )
 
 
-  def _DefaultParameters( self ):
+  def _DefaultParameters( self, request_data ):
     """ Some very common request parameters """
-    line, column = vimsupport.CurrentLineAndColumn()
     parameters = {}
-    parameters[ 'line' ], parameters[ 'column' ] = line + 1, column + 1
+    parameters[ 'line' ] = request_data[ 'line_num' ] + 1
-    parameters[ 'buffer' ] = '\n'.join( vim.current.buffer )
+    parameters[ 'column' ] = request_data[ 'column_num' ] + 1
-    parameters[ 'filename' ] = vim.current.buffer.name
+    filepath = request_data[ 'filepath' ]
+    parameters[ 'buffer' ] = request_data[ 'file_data' ][ filepath ][
+      'contents' ]
+    parameters[ 'filename' ] = filepath
     return parameters
 
 
   def _ServerIsRunning( self ):
     """ Check if our OmniSharp server is running """
-    return  ( self._omnisharp_port != None and
-        self._GetResponse( '/checkalivestatus', silent=True ) != None )
-
-
-  def _FindFreePort( self ):
-    """ Find port without an OmniSharp server running on it """
-    port = int( vimsupport.GetVariableValue(
-        'g:ycm_csharp_server_port' ) )
-    while self._GetResponse( '/checkalivestatus',
-        silent=True,
-        port=port ) != None:
-      port += 1
-    return port
-
-
-  def _PortToHost( self, port=None ):
-    if port == None:
-      port = self._omnisharp_port
-    return 'http://localhost:' + str( port )
-
-
-  def _GetResponse( self, endPoint, parameters={}, silent=False, port=None ):
-    """ Handle communication with server """
-    target = urlparse.urljoin( self._PortToHost( port ), endPoint )
-    parameters = urllib.urlencode( parameters )
     try:
-      response = urllib2.urlopen( target, parameters )
+      return bool( self._omnisharp_port and
-      return json.loads( response.read() )
+                  self._GetResponse( '/checkalivestatus', silent = True ) )
-    except Exception:
+    except:
-      # TODO: Add logging for this case. We can't post a Vim message because Vim
+      return False
-      # crashes when that's done from a no-GUI thread.
-      return None
 
 
-def _FindSolutionFiles():
+  def _ServerLocation( self ):
+    return 'http://localhost:' + str( self._omnisharp_port )
+
+
+  def _GetResponse( self, handler, parameters = {}, silent = False ):
+    """ Handle communication with server """
+    # TODO: Replace usage of urllib with Requests
+    target = urlparse.urljoin( self._ServerLocation(), handler )
+    parameters = urllib.urlencode( parameters )
+    response = urllib2.urlopen( target, parameters )
+    return json.loads( response.read() )
+
+
+def _FindSolutionFiles( filepath ):
   """ Find solution files by searching upwards in the file tree """
-  folder = os.path.dirname( vim.current.buffer.name )
+  folder = os.path.dirname( filepath )
   solutionfiles = glob.glob1( folder, '*.sln' )
   while not solutionfiles:
     lastfolder = folder

python/ycm/completers/cs/hook.py

@@ -17,5 +17,5 @@
 
 from ycm.completers.cs.cs_completer import CsharpCompleter
 
-def GetCompleter():
+def GetCompleter( user_options ):
-    return CsharpCompleter()
+  return CsharpCompleter( user_options )

python/ycm/completers/general/filename_completer.py

@@ -16,26 +16,21 @@
 # You should have received a copy of the GNU General Public License
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
-import vim
 import os
 import re
 
-from ycm import vimsupport
+from ycm.completers.completer import Completer
-from ycm.completers.threaded_completer import ThreadedCompleter
 from ycm.completers.cpp.clang_completer import InCFamilyFile
 from ycm.completers.cpp.flags import Flags
+from ycm.server import responses
 
-USE_WORKING_DIR = vimsupport.GetBoolValue(
+class FilenameCompleter( Completer ):
-  'g:ycm_filepath_completion_use_working_dir' )
-
-
-class FilenameCompleter( ThreadedCompleter ):
   """
   General completer that provides filename and filepath completions.
   """
 
-  def __init__( self ):
+  def __init__( self, user_options ):
-    super( FilenameCompleter, self ).__init__()
+    super( FilenameCompleter, self ).__init__( user_options )
     self._flags = Flags()
 
     self._path_regex = re.compile( """
@@ -57,25 +52,35 @@ class FilenameCompleter( ThreadedCompleter ):
     self._include_regex = re.compile( include_regex_common )
 
 
-  def AtIncludeStatementStart( self, start_column ):
+  def AtIncludeStatementStart( self, request_data ):
-    return ( InCFamilyFile() and
+    start_column = request_data[ 'start_column' ]
+    current_line = request_data[ 'line_value' ]
+    filepath = request_data[ 'filepath' ]
+    filetypes = request_data[ 'file_data' ][ filepath ][ 'filetypes' ]
+    return ( InCFamilyFile( filetypes ) and
              self._include_start_regex.match(
-               vim.current.line[ :start_column ] ) )
+               current_line[ :start_column ] ) )
 
 
-  def ShouldUseNowInner( self, start_column ):
+  def ShouldUseNowInner( self, request_data ):
-    return ( start_column and ( vim.current.line[ start_column - 1 ] == '/' or
+    start_column = request_data[ 'start_column' ]
-             self.AtIncludeStatementStart( start_column ) ) )
+    current_line = request_data[ 'line_value' ]
+    return ( start_column and ( current_line[ start_column - 1 ] == '/' or
+             self.AtIncludeStatementStart( request_data ) ) )
 
 
   def SupportedFiletypes( self ):
     return []
 
 
-  def ComputeCandidates( self, unused_query, start_column ):
+  def ComputeCandidatesInner( self, request_data ):
-    line = vim.current.line[ :start_column ]
+    current_line = request_data[ 'line_value' ]
+    start_column = request_data[ 'start_column' ]
+    filepath = request_data[ 'filepath' ]
+    filetypes = request_data[ 'file_data' ][ filepath ][ 'filetypes' ]
+    line = current_line[ :start_column ]
 
-    if InCFamilyFile():
+    if InCFamilyFile( filetypes ):
       include_match = self._include_regex.search( line )
       if include_match:
         path_dir = line[ include_match.end(): ]
@@ -83,20 +88,26 @@ class FilenameCompleter( ThreadedCompleter ):
         # http://gcc.gnu.org/onlinedocs/cpp/Include-Syntax.html
         include_current_file_dir = '<' not in include_match.group()
         return _GenerateCandidatesForPaths(
-          self.GetPathsIncludeCase( path_dir, include_current_file_dir ) )
+          self.GetPathsIncludeCase( path_dir,
+                                    include_current_file_dir,
+                                    filepath ) )
 
     path_match = self._path_regex.search( line )
     path_dir = os.path.expanduser( path_match.group() ) if path_match else ''
 
-    return _GenerateCandidatesForPaths( _GetPathsStandardCase( path_dir ) )
+    return _GenerateCandidatesForPaths(
+      _GetPathsStandardCase(
+        path_dir,
+        self.user_options[ 'filepath_completion_use_working_dir' ],
+        filepath ) )
 
 
-  def GetPathsIncludeCase( self, path_dir, include_current_file_dir ):
+  def GetPathsIncludeCase( self, path_dir, include_current_file_dir, filepath ):
     paths = []
-    include_paths = self._flags.UserIncludePaths( vim.current.buffer.name )
+    include_paths = self._flags.UserIncludePaths( filepath )
 
     if include_current_file_dir:
-      include_paths.append( os.path.dirname( vim.current.buffer.name ) )
+      include_paths.append( os.path.dirname( filepath ) )
 
     for include_path in include_paths:
       try:
@@ -110,9 +121,9 @@ class FilenameCompleter( ThreadedCompleter ):
     return sorted( set( paths ) )
 
 
-def _GetPathsStandardCase( path_dir ):
+def _GetPathsStandardCase( path_dir, use_working_dir, filepath ):
-  if not USE_WORKING_DIR and not path_dir.startswith( '/' ):
+  if not use_working_dir and not path_dir.startswith( '/' ):
-    path_dir = os.path.join( os.path.dirname( vim.current.buffer.name ),
+    path_dir = os.path.join( os.path.dirname( filepath ),
                              path_dir )
 
   try:
@@ -135,8 +146,8 @@ def _GenerateCandidatesForPaths( absolute_paths ):
     seen_basenames.add( basename )
 
     is_dir = os.path.isdir( absolute_path )
-    completion_dicts.append( { 'word': basename,
+    completion_dicts.append(
-                               'dup': 1,
+      responses.BuildCompletionData( basename,
-                               'menu': '[Dir]' if is_dir else '[File]' } )
+                                     '[Dir]' if is_dir else '[File]' ) )
 
   return completion_dicts

python/ycm/completers/general/general_completer_store.py

@@ -21,13 +21,7 @@
 from ycm.completers.completer import Completer
 from ycm.completers.all.identifier_completer import IdentifierCompleter
 from ycm.completers.general.filename_completer import FilenameCompleter
-
+from ycm.completers.general.ultisnips_completer import UltiSnipsCompleter
-try:
-  from ycm.completers.general.ultisnips_completer import UltiSnipsCompleter
-  USE_ULTISNIPS_COMPLETER = True
-except ImportError:
-  USE_ULTISNIPS_COMPLETER = False
-
 
 
 class GeneralCompleterStore( Completer ):
@@ -38,12 +32,11 @@ class GeneralCompleterStore( Completer ):
   GeneralCompleterStore are passed to all general completers.
   """
 
-  def __init__( self ):
+  def __init__( self, user_options ):
-    super( GeneralCompleterStore, self ).__init__()
+    super( GeneralCompleterStore, self ).__init__( user_options )
-    self._identifier_completer = IdentifierCompleter()
+    self._identifier_completer = IdentifierCompleter( user_options )
-    self._filename_completer = FilenameCompleter()
+    self._filename_completer = FilenameCompleter( user_options )
-    self._ultisnips_completer = ( UltiSnipsCompleter()
+    self._ultisnips_completer = UltiSnipsCompleter( user_options )
-                                  if USE_ULTISNIPS_COMPLETER else None )
     self._non_filename_completers = filter( lambda x: x,
                                             [ self._ultisnips_completer,
                                               self._identifier_completer ] )
@@ -58,17 +51,21 @@ class GeneralCompleterStore( Completer ):
     return set()
 
 
-  def ShouldUseNow( self, start_column ):
+  def GetIdentifierCompleter( self ):
+    return self._identifier_completer
+
+
+  def ShouldUseNow( self, request_data ):
     self._current_query_completers = []
 
-    if self._filename_completer.ShouldUseNow( start_column ):
+    if self._filename_completer.ShouldUseNow( request_data ):
       self._current_query_completers = [ self._filename_completer ]
       return True
 
     should_use_now = False
 
     for completer in self._non_filename_completers:
-      should_use_this_completer = completer.ShouldUseNow( start_column )
+      should_use_this_completer = completer.ShouldUseNow( request_data )
       should_use_now = should_use_now or should_use_this_completer
 
       if should_use_this_completer:
@@ -77,69 +74,49 @@ class GeneralCompleterStore( Completer ):
     return should_use_now
 
 
-  def CandidatesForQueryAsync( self, query, start_column ):
+  def ComputeCandidates( self, request_data ):
-    for completer in self._current_query_completers:
+    if not self.ShouldUseNow( request_data ):
-      completer.CandidatesForQueryAsync( query, start_column )
+      return []
 
-
-  def AsyncCandidateRequestReady( self ):
-    return all( x.AsyncCandidateRequestReady() for x in
-                self._current_query_completers )
-
-
-  def CandidatesFromStoredRequest( self ):
     candidates = []
     for completer in self._current_query_completers:
-      candidates += completer.CandidatesFromStoredRequest()
+      candidates += completer.ComputeCandidates( request_data )
 
     return candidates
 
 
-  def OnFileReadyToParse( self ):
+  def OnFileReadyToParse( self, request_data ):
     for completer in self._all_completers:
-      completer.OnFileReadyToParse()
+      completer.OnFileReadyToParse( request_data )
 
 
-  def OnCursorMovedInsertMode( self ):
+  def OnBufferVisit( self, request_data ):
     for completer in self._all_completers:
-      completer.OnCursorMovedInsertMode()
+      completer.OnBufferVisit( request_data )
 
 
-  def OnCursorMovedNormalMode( self ):
+  def OnBufferUnload( self, request_data ):
     for completer in self._all_completers:
-      completer.OnCursorMovedNormalMode()
+      completer.OnBufferUnload( request_data )
 
 
-  def OnBufferVisit( self ):
+  def OnInsertLeave( self, request_data ):
     for completer in self._all_completers:
-      completer.OnBufferVisit()
+      completer.OnInsertLeave( request_data )
 
 
-  def OnBufferUnload( self, deleted_buffer_file ):
+  def OnCurrentIdentifierFinished( self, request_data ):
     for completer in self._all_completers:
-      completer.OnBufferUnload( deleted_buffer_file )
+      completer.OnCurrentIdentifierFinished( request_data )
-
-
-  def OnCursorHold( self ):
-    for completer in self._all_completers:
-      completer.OnCursorHold()
-
-
-  def OnInsertLeave( self ):
-    for completer in self._all_completers:
-      completer.OnInsertLeave()
-
-
-  def OnVimLeave( self ):
-    for completer in self._all_completers:
-      completer.OnVimLeave()
-
-
-  def OnCurrentIdentifierFinished( self ):
-    for completer in self._all_completers:
-      completer.OnCurrentIdentifierFinished()
 
 
   def GettingCompletions( self ):
     for completer in self._all_completers:
       completer.GettingCompletions()
+
+
+  def Shutdown( self ):
+    for completer in self._all_completers:
+      completer.Shutdown()
+
+

python/ycm/completers/general/ultisnips_completer.py

@@ -19,7 +19,7 @@
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
 from ycm.completers.general_completer import GeneralCompleter
-from UltiSnips import UltiSnips_Manager
+from ycm.server import responses
 
 
 class UltiSnipsCompleter( GeneralCompleter ):
@@ -27,42 +27,28 @@ class UltiSnipsCompleter( GeneralCompleter ):
   General completer that provides UltiSnips snippet names in completions.
   """
 
-  def __init__( self ):
+  def __init__( self, user_options ):
-    super( UltiSnipsCompleter, self ).__init__()
+    super( UltiSnipsCompleter, self ).__init__( user_options )
     self._candidates = None
     self._filtered_candidates = None
 
 
-  def ShouldUseNowInner( self, start_column ):
+  def ShouldUseNow( self, request_data ):
-    return self.QueryLengthAboveMinThreshold( start_column )
+    return self.QueryLengthAboveMinThreshold( request_data )
 
 
-  def CandidatesForQueryAsync( self, query, unused_start_column ):
+  def ComputeCandidates( self, request_data ):
-    self._filtered_candidates = self.FilterAndSortCandidates( self._candidates,
+    if not self.ShouldUseNow( request_data ):
-                                                              query )
+      return []
+    return self.FilterAndSortCandidates(
+      self._candidates, request_data[ 'query' ] )
 
 
-  def AsyncCandidateRequestReady( self ):
+  def OnBufferVisit( self, request_data ):
-    return True
+    raw_candidates = request_data.get( 'ultisnips_snippets', [] )
+    self._candidates = [
+        responses.BuildCompletionData(
+            str( snip[ 'trigger' ] ),
+            str( '<snip> ' + snip[ 'description' ].encode( 'utf-8' ) ) )
+        for snip in raw_candidates ]
 
-
-  def CandidatesFromStoredRequest( self ):
-    return self._filtered_candidates if self._filtered_candidates else []
-
-
-  def OnBufferVisit( self ):
-    self._candidates = _GetCandidates()
-
-
-def _GetCandidates():
-  try:
-    rawsnips = UltiSnips_Manager._snips( '', 1 )
-
-    # UltiSnips_Manager._snips() returns a class instance where:
-    # class.trigger - name of snippet trigger word ( e.g. defn or testcase )
-    # class.description - description of the snippet
-    return  [ { 'word': str( snip.trigger ),
-                'menu': str( '<snip> ' + snip.description.encode('utf-8') ) }
-              for snip in rawsnips ]
-  except:
-    return []

python/ycm/completers/general_completer.py

@@ -29,8 +29,8 @@ class GeneralCompleter( Completer ):
   Subclass Completer directly.
 
   """
-  def __init__( self ):
+  def __init__( self, user_options ):
-    super( GeneralCompleter, self ).__init__()
+    super( GeneralCompleter, self ).__init__( user_options )
 
 
   def SupportedFiletypes( self ):

python/ycm/completers/objc/hook.py

@@ -20,8 +20,8 @@
 import ycm_core
 from ycm.completers.cpp.clang_completer import ClangCompleter
 
-def GetCompleter():
+def GetCompleter( user_options ):
   if ycm_core.HasClangSupport():
-    return ClangCompleter()
+    return ClangCompleter( user_options )
   else:
     return None

python/ycm/completers/objcpp/hook.py

@@ -20,9 +20,9 @@
 import ycm_core
 from ycm.completers.cpp.clang_completer import ClangCompleter
 
-def GetCompleter():
+def GetCompleter( user_options ):
   if ycm_core.HasClangSupport():
-    return ClangCompleter()
+    return ClangCompleter( user_options )
   else:
     return None
 

python/ycm/completers/python/hook.py

@@ -17,5 +17,5 @@
 
 from ycm.completers.python.jedi_completer import JediCompleter
 
-def GetCompleter():
+def GetCompleter( user_options ):
-    return JediCompleter()
+  return JediCompleter( user_options )

python/ycm/completers/python/jedi

@@ -1 +0,0 @@
-Subproject commit d5d12716b1d67df9cbaa4d3ea0c90e47c0023208

python/ycm/completers/python/jedi_completer.py

@@ -19,34 +19,25 @@
 # You should have received a copy of the GNU General Public License
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
-import vim
+from ycm.completers.completer import Completer
-from ycm.completers.threaded_completer import ThreadedCompleter
+from ycm.server import responses
-from ycm import vimsupport
 
-import sys
-from os.path import join, abspath, dirname
-
-# We need to add the jedi package to sys.path, but it's important that we clean
-# up after ourselves, because ycm.YouCompletMe.GetFiletypeCompleterForFiletype
-# removes sys.path[0] after importing completers.python.hook
-sys.path.insert( 0, join( abspath( dirname( __file__ ) ), 'jedi' ) )
 try:
   import jedi
 except ImportError:
-  vimsupport.PostVimMessage(
+  raise ImportError(
     'Error importing jedi. Make sure the jedi submodule has been checked out. '
     'In the YouCompleteMe folder, run "git submodule update --init --recursive"')
-sys.path.pop( 0 )
 
 
-class JediCompleter( ThreadedCompleter ):
+class JediCompleter( Completer ):
   """
   A Completer that uses the Jedi completion engine.
   https://jedi.readthedocs.org/en/latest/
   """
 
-  def __init__( self ):
+  def __init__( self, user_options ):
-    super( JediCompleter, self ).__init__()
+    super( JediCompleter, self ).__init__( user_options )
 
 
   def SupportedFiletypes( self ):
@@ -54,113 +45,109 @@ class JediCompleter( ThreadedCompleter ):
     return [ 'python' ]
 
 
-  def _GetJediScript( self ):
+  def _GetJediScript( self, request_data ):
-      contents = '\n'.join( vim.current.buffer )
+      filename = request_data[ 'filepath' ]
-      line, column = vimsupport.CurrentLineAndColumn()
+      contents = request_data[ 'file_data' ][ filename ][ 'contents' ]
       # Jedi expects lines to start at 1, not 0
-      line += 1
+      line = request_data[ 'line_num' ] + 1
-      filename = vim.current.buffer.name
+      column = request_data[ 'column_num' ]
 
       return jedi.Script( contents, line, column, filename )
 
 
-  def ComputeCandidates( self, unused_query, unused_start_column ):
+  def ComputeCandidatesInner( self, request_data ):
-    script = self._GetJediScript()
+    script = self._GetJediScript( request_data )
-
+    return [ responses.BuildCompletionData(
-    return [ { 'word': str( completion.name ),
+                str( completion.name ),
-               'menu': str( completion.description ),
+                str( completion.description ),
-               'info': str( completion.doc ) }
+                str( completion.doc ) )
              for completion in script.completions() ]
 
-
   def DefinedSubcommands( self ):
-    return [ "GoToDefinition",
+    return [ 'GoToDefinition',
-             "GoToDeclaration",
+             'GoToDeclaration',
-             "GoToDefinitionElseDeclaration" ]
+             'GoToDefinitionElseDeclaration' ]
 
 
-  def OnUserCommand( self, arguments ):
+  def OnUserCommand( self, arguments, request_data ):
     if not arguments:
-      self.EchoUserCommandsHelpMessage()
+      raise ValueError( self.UserCommandsHelpMessage() )
-      return
 
     command = arguments[ 0 ]
     if command == 'GoToDefinition':
-      self._GoToDefinition()
+      return self._GoToDefinition( request_data )
     elif command == 'GoToDeclaration':
-      self._GoToDeclaration()
+      return self._GoToDeclaration( request_data )
     elif command == 'GoToDefinitionElseDeclaration':
-      self._GoToDefinitionElseDeclaration()
+      return self._GoToDefinitionElseDeclaration( request_data )
+    raise ValueError( self.UserCommandsHelpMessage() )
 
 
-  def _GoToDefinition( self ):
+  def _GoToDefinition( self, request_data ):
-    definitions = self._GetDefinitionsList()
+    definitions = self._GetDefinitionsList( request_data )
     if definitions:
-      self._JumpToLocation( definitions )
+      return self._BuildGoToResponse( definitions )
     else:
-      vimsupport.PostVimMessage( 'Can\'t jump to definition.' )
+      raise RuntimeError( 'Can\'t jump to definition.' )
 
 
-  def _GoToDeclaration( self ):
+  def _GoToDeclaration( self, request_data ):
-    definitions = self._GetDefinitionsList( declaration = True )
+    definitions = self._GetDefinitionsList( request_data, declaration = True )
     if definitions:
-      self._JumpToLocation( definitions )
+      return self._BuildGoToResponse( definitions )
     else:
-      vimsupport.PostVimMessage( 'Can\'t jump to declaration.' )
+      raise RuntimeError( 'Can\'t jump to declaration.' )
 
 
-  def _GoToDefinitionElseDeclaration( self ):
+  def _GoToDefinitionElseDeclaration( self, request_data ):
-    definitions = self._GetDefinitionsList() or \
+    definitions = ( self._GetDefinitionsList( request_data ) or
-        self._GetDefinitionsList( declaration = True )
+        self._GetDefinitionsList( request_data, declaration = True ) )
     if definitions:
-      self._JumpToLocation( definitions )
+      return self._BuildGoToResponse( definitions )
     else:
-      vimsupport.PostVimMessage( 'Can\'t jump to definition or declaration.' )
+      raise RuntimeError( 'Can\'t jump to definition or declaration.' )
 
 
-  def _GetDefinitionsList( self, declaration = False ):
+  def _GetDefinitionsList( self, request_data, declaration = False ):
     definitions = []
-    script = self._GetJediScript()
+    script = self._GetJediScript( request_data )
     try:
       if declaration:
         definitions = script.goto_definitions()
       else:
         definitions = script.goto_assignments()
     except jedi.NotFoundError:
-      vimsupport.PostVimMessage(
+      raise RuntimeError(
-                  "Cannot follow nothing. Put your cursor on a valid name." )
+                  'Cannot follow nothing. Put your cursor on a valid name.' )
-    except Exception as e:
-      vimsupport.PostVimMessage(
-                  "Caught exception, aborting. Full error: " + str( e ) )
 
     return definitions
 
 
-  def _JumpToLocation( self, definition_list ):
+  def _BuildGoToResponse( self, definition_list ):
     if len( definition_list ) == 1:
       definition = definition_list[ 0 ]
       if definition.in_builtin_module():
         if definition.is_keyword:
-          vimsupport.PostVimMessage(
+          raise RuntimeError(
-                  "Cannot get the definition of Python keywords." )
+                  'Cannot get the definition of Python keywords.' )
         else:
-          vimsupport.PostVimMessage( "Builtin modules cannot be displayed." )
+          raise RuntimeError( 'Builtin modules cannot be displayed.' )
       else:
-        vimsupport.JumpToLocation( definition.module_path,
+        return responses.BuildGoToResponse( definition.module_path,
-                                   definition.line,
+                                            definition.line - 1,
-                                   definition.column + 1 )
+                                            definition.column )
     else:
       # multiple definitions
       defs = []
       for definition in definition_list:
         if definition.in_builtin_module():
-          defs.append( {'text': 'Builtin ' + \
+          defs.append( responses.BuildDescriptionOnlyGoToResponse(
-                       definition.description.encode( 'utf-8' ) } )
+                       'Builtin ' + definition.description ) )
         else:
-          defs.append( {'filename': definition.module_path.encode( 'utf-8' ),
+          defs.append(
-                        'lnum': definition.line,
+            responses.BuildGoToResponse( definition.module_path,
-                        'col': definition.column + 1,
+                                         definition.line - 1,
-                        'text': definition.description.encode( 'utf-8' ) } )
+                                         definition.column,
+                                         definition.description ) )
+      return defs
 
-      vim.eval( 'setqflist( %s )' % repr( defs ) )
-      vim.eval( 'youcompleteme#OpenGoToList()' )

python/ycm/completers/threaded_completer.py

@@ -1,102 +0,0 @@
-#!/usr/bin/env python
-#
-# Copyright (C) 2011, 2012  Strahinja Val Markovic  <val@markovic.io>
-#
-# This file is part of YouCompleteMe.
-#
-# YouCompleteMe is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# YouCompleteMe is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
-
-import abc
-from threading import Thread, Event
-from ycm.completers.completer import Completer
-
-class ThreadedCompleter( Completer ):
-  """A subclass of Completer that abstracts away the use of a background thread.
-
-  This is a great class to subclass for your custom completer. It will provide
-  you with async computation of candidates when you implement the
-  ComputeCandidates() method; no need to worry about threads, locks, events or
-  similar.
-
-  If you use this class as the base class for your Completer, you DON'T need to
-  provide implementations for the CandidatesForQueryAsync(),
-  AsyncCandidateRequestReady() and CandidatesFromStoredRequest() functions. Just
-  implement ComputeCandidates().
-
-  For examples of subclasses of this class, see the following files:
-  python/completers/general/filename_completer.py
-  """
-
-  def __init__( self ):
-    super( ThreadedCompleter, self ).__init__()
-    self._query_ready = Event()
-    self._candidates_ready = Event()
-    self._candidates = None
-    self._start_completion_thread()
-
-
-  def _start_completion_thread( self ):
-    self._completion_thread = Thread( target=self.SetCandidates )
-    self._completion_thread.daemon = True
-    self._completion_thread.start()
-
-
-  def CandidatesForQueryAsyncInner( self, query, start_column ):
-    self._candidates = None
-    self._candidates_ready.clear()
-    self._query = query
-    self._start_column = start_column
-    self._query_ready.set()
-
-
-  def AsyncCandidateRequestReadyInner( self ):
-    return WaitAndClearIfSet( self._candidates_ready, timeout=0.005 )
-
-
-  def CandidatesFromStoredRequestInner( self ):
-    return self._candidates or []
-
-
-  @abc.abstractmethod
-  def ComputeCandidates( self, query, start_column ):
-    """This function should compute the candidates to show to the user.
-    The return value should be of the same type as that for
-    CandidatesFromStoredRequest()."""
-    pass
-
-
-  def SetCandidates( self ):
-    while True:
-      try:
-        WaitAndClearIfSet( self._query_ready )
-        self._candidates = self.ComputeCandidates( self._query,
-                                                   self._start_column )
-      except:
-        self._query_ready.clear()
-        self._candidates = []
-      self._candidates_ready.set()
-
-
-def WaitAndClearIfSet( event, timeout=None ):
-  """Given an |event| and a |timeout|, waits for the event a maximum of timeout
-  seconds. After waiting, clears the event if it's set and returns the state of
-  the event before it was cleared."""
-
-  # We can't just do flag_is_set = event.wait( timeout ) because that breaks on
-  # Python 2.6
-  event.wait( timeout )
-  flag_is_set = event.is_set()
-  if flag_is_set:
-      event.clear()
-  return flag_is_set

python/ycm/extra_conf_store.py

@@ -24,25 +24,30 @@ import imp
 import random
 import string
 import sys
-import vim
+import logging
-from ycm import vimsupport
+from threading import Lock
+from ycm import user_options_store
+from ycm.server.responses import UnknownExtraConf
 from fnmatch import fnmatch
 
 # Constants
 YCM_EXTRA_CONF_FILENAME = '.ycm_extra_conf.py'
-CONFIRM_CONF_FILE_MESSAGE = ('Found {0}. Load? \n\n(Question can be turned '
-                             'off with options, see YCM docs)')
-GLOBAL_YCM_EXTRA_CONF_FILE = os.path.expanduser(
-    vimsupport.GetVariableValue( "g:ycm_global_ycm_extra_conf" )
-)
 
 # Singleton variables
 _module_for_module_file = {}
+_module_for_module_file_lock = Lock()
 _module_file_for_source_file = {}
+_module_file_for_source_file_lock = Lock()
+
+
+def Reset():
+  global _module_for_module_file, _module_file_for_source_file
+  _module_for_module_file = {}
+  _module_file_for_source_file = {}
 
 
 def ModuleForSourceFile( filename ):
-  return _Load( ModuleFileForSourceFile( filename ) )
+  return Load( ModuleFileForSourceFile( filename ) )
 
 
 def ModuleFileForSourceFile( filename ):
@@ -50,35 +55,50 @@ def ModuleFileForSourceFile( filename ):
   order and return the filename of the first module that was allowed to load.
   If no module was found or allowed to load, None is returned."""
 
-  if not filename in _module_file_for_source_file:
+  with _module_file_for_source_file_lock:
-    for module_file in _ExtraConfModuleSourceFilesForFile( filename ):
+    if not filename in _module_file_for_source_file:
-      if _Load( module_file ):
+      for module_file in _ExtraConfModuleSourceFilesForFile( filename ):
-        _module_file_for_source_file[ filename ] = module_file
+        if Load( module_file ):
-        break
+          _module_file_for_source_file[ filename ] = module_file
+          break
 
   return _module_file_for_source_file.setdefault( filename )
 
 
-def CallExtraConfYcmCorePreloadIfExists():
+def CallGlobalExtraConfYcmCorePreloadIfExists():
-  _CallExtraConfMethod( 'YcmCorePreload' )
+  _CallGlobalExtraConfMethod( 'YcmCorePreload' )
 
 
-def CallExtraConfVimCloseIfExists():
+def Shutdown():
-  _CallExtraConfMethod( 'VimClose' )
+  # VimClose is for the sake of backwards compatibility; it's a no-op when it
+  # doesn't exist.
+  _CallGlobalExtraConfMethod( 'VimClose' )
+  _CallGlobalExtraConfMethod( 'Shutdown' )
 
 
-def _CallExtraConfMethod( function_name ):
+def _CallGlobalExtraConfMethod( function_name ):
-  vim_current_working_directory = vim.eval( 'getcwd()' )
+  logger = _Logger()
-  path_to_dummy = os.path.join( vim_current_working_directory, 'DUMMY_FILE' )
+  global_ycm_extra_conf = _GlobalYcmExtraConfFileLocation()
-  module = ModuleForSourceFile( path_to_dummy )
+  if not ( global_ycm_extra_conf and
-  if not module or not hasattr( module, function_name ):
+           os.path.exists( global_ycm_extra_conf ) ):
+    logger.debug( 'No global extra conf, not calling method ' + function_name )
     return
+
+  module = Load( global_ycm_extra_conf, force = True )
+  if not module or not hasattr( module, function_name ):
+    logger.debug( 'Global extra conf not loaded or no function ' +
+                  function_name )
+    return
+
+  logger.info( 'Calling global extra conf method {0} on conf file {1}'.format(
+      function_name, global_ycm_extra_conf ) )
   getattr( module, function_name )()
 
 
 def _Disable( module_file ):
   """Disables the loading of a module for the current session."""
-  _module_for_module_file[ module_file ] = None
+  with _module_for_module_file_lock:
+    _module_for_module_file[ module_file ] = None
 
 
 def _ShouldLoad( module_file ):
@@ -86,20 +106,25 @@ def _ShouldLoad( module_file ):
   decide using a white-/blacklist and ask the user for confirmation as a
   fallback."""
 
-  if ( module_file == GLOBAL_YCM_EXTRA_CONF_FILE or
+  if ( module_file == _GlobalYcmExtraConfFileLocation() or
-        not vimsupport.GetBoolValue( 'g:ycm_confirm_extra_conf' ) ):
+       not user_options_store.Value( 'confirm_extra_conf' ) ):
     return True
 
-  globlist = vimsupport.GetVariableValue( 'g:ycm_extra_conf_globlist' )
+  globlist = user_options_store.Value( 'extra_conf_globlist' )
   for glob in globlist:
     is_blacklisted = glob[0] == '!'
     if _MatchesGlobPattern( module_file, glob.lstrip('!') ):
       return not is_blacklisted
 
-  return vimsupport.Confirm( CONFIRM_CONF_FILE_MESSAGE.format( module_file ) )
+  # We disable the file if it's unknown so that we don't ask the user about it
+  # repeatedly. Raising UnknownExtraConf should result in the client sending
+  # another request to load the module file if the user explicitly chooses to do
+  # that.
+  _Disable( module_file )
+  raise UnknownExtraConf( module_file )
 
 
-def _Load( module_file, force = False ):
+def Load( module_file, force = False ):
   """Load and return the module contained in a file.
   Using force = True the module will be loaded regardless
   of the criteria in _ShouldLoad.
@@ -109,11 +134,13 @@ def _Load( module_file, force = False ):
     return None
 
   if not force:
-    if module_file in _module_for_module_file:
+    with _module_for_module_file_lock:
-      return _module_for_module_file[ module_file ]
+      if module_file in _module_for_module_file:
+        return _module_for_module_file[ module_file ]
 
     if not _ShouldLoad( module_file ):
-      return _Disable( module_file )
+      _Disable( module_file )
+      return None
 
   # This has to be here because a long time ago, the ycm_extra_conf.py files
   # used to import clang_helpers.py from the cpp folder. This is not needed
@@ -123,7 +150,8 @@ def _Load( module_file, force = False ):
   module = imp.load_source( _RandomName(), module_file )
   del sys.path[ 0 ]
 
-  _module_for_module_file[ module_file ] = module
+  with _module_for_module_file_lock:
+    _module_for_module_file[ module_file ] = module
   return module
 
 
@@ -139,15 +167,16 @@ def _MatchesGlobPattern( filename, glob ):
 def _ExtraConfModuleSourceFilesForFile( filename ):
   """For a given filename, search all parent folders for YCM_EXTRA_CONF_FILENAME
   files that will compute the flags necessary to compile the file.
-  If GLOBAL_YCM_EXTRA_CONF_FILE exists it is returned as a fallback."""
+  If _GlobalYcmExtraConfFileLocation() exists it is returned as a fallback."""
 
   for folder in _PathsToAllParentFolders( filename ):
     candidate = os.path.join( folder, YCM_EXTRA_CONF_FILENAME )
     if os.path.exists( candidate ):
       yield candidate
-  if ( GLOBAL_YCM_EXTRA_CONF_FILE
+  global_ycm_extra_conf = _GlobalYcmExtraConfFileLocation()
-       and os.path.exists( GLOBAL_YCM_EXTRA_CONF_FILE ) ):
+  if ( global_ycm_extra_conf
-    yield GLOBAL_YCM_EXTRA_CONF_FILE
+       and os.path.exists( global_ycm_extra_conf ) ):
+    yield global_ycm_extra_conf
 
 
 def _PathsToAllParentFolders( filename ):
@@ -188,3 +217,12 @@ def _DirectoryOfThisScript():
 def _RandomName():
   """Generates a random module name."""
   return ''.join( random.choice( string.ascii_lowercase ) for x in range( 15 ) )
+
+
+def _GlobalYcmExtraConfFileLocation():
+  return os.path.expanduser(
+    user_options_store.Value( 'global_ycm_extra_conf' ) )
+
+
+def _Logger():
+  return logging.getLogger( __name__ )

python/ycm/server/default_settings.json

@@ -0,0 +1,32 @@
+{
+  "filepath_completion_use_working_dir": 0,
+  "min_num_of_chars_for_completion": 2,
+  "min_num_identifier_candidate_chars": 0,
+  "semantic_triggers": {},
+  "filetype_specific_completion_to_disable": {
+    "gitcommit": 1
+  },
+  "seed_identifiers_with_syntax": 0,
+  "collect_identifiers_from_comments_and_strings": 0,
+  "collect_identifiers_from_tags_files": 0,
+  "extra_conf_globlist": [],
+  "global_ycm_extra_conf": "",
+  "confirm_extra_conf": 1,
+  "complete_in_comments": 0,
+  "complete_in_strings": 1,
+  "max_diagnostics_to_display": 30,
+  "filetype_whitelist": {
+    "*": 1
+  },
+  "filetype_blacklist": {
+    "tagbar": 1,
+    "qf": 1,
+    "notes": 1,
+    "markdown": 1,
+    "unite": 1,
+    "text": 1
+  },
+  "auto_start_csharp_server": 1,
+  "auto_stop_csharp_server": 1,
+  "csharp_server_port": 2000
+}

python/ycm/server/handlers.py

@@ -0,0 +1,222 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+from os import path
+
+try:
+  import ycm_core
+except ImportError as e:
+  raise RuntimeError(
+    'Error importing ycm_core. Are you sure you have placed a '
+    'version 3.2+ libclang.[so|dll|dylib] in folder "{0}"? '
+    'See the Installation Guide in the docs. Full error: {1}'.format(
+      path.realpath( path.join( path.abspath( __file__ ), '../../..' ) ),
+      str( e ) ) )
+
+import atexit
+import logging
+import json
+import bottle
+import httplib
+from bottle import request, response
+import server_state
+from ycm import user_options_store
+from ycm.server.responses import BuildExceptionResponse
+from ycm import extra_conf_store
+
+
+# num bytes for the request body buffer; request.json only works if the request
+# size is less than this
+bottle.Request.MEMFILE_MAX = 300 * 1024
+
+# TODO: rename these to _lower_case
+SERVER_STATE = None
+LOGGER = logging.getLogger( __name__ )
+app = bottle.Bottle()
+
+
+@app.post( '/event_notification' )
+def EventNotification():
+  LOGGER.info( 'Received event notification' )
+  request_data = request.json
+  event_name = request_data[ 'event_name' ]
+  LOGGER.debug( 'Event name: %s', event_name )
+
+  event_handler = 'On' + event_name
+  getattr( SERVER_STATE.GetGeneralCompleter(), event_handler )( request_data )
+
+  filetypes = request_data[ 'filetypes' ]
+  response_data = None
+  if SERVER_STATE.FiletypeCompletionUsable( filetypes ):
+    response_data = getattr( SERVER_STATE.GetFiletypeCompleter( filetypes ),
+                             event_handler )( request_data )
+
+  if response_data:
+    return _JsonResponse( response_data )
+
+
+@app.post( '/run_completer_command' )
+def RunCompleterCommand():
+  LOGGER.info( 'Received command request' )
+  request_data = request.json
+  completer = _GetCompleterForRequestData( request_data )
+
+  return _JsonResponse( completer.OnUserCommand(
+      request_data[ 'command_arguments' ],
+      request_data ) )
+
+
+@app.post( '/completions' )
+def GetCompletions():
+  LOGGER.info( 'Received completion request' )
+  request_data = request.json
+  do_filetype_completion = SERVER_STATE.ShouldUseFiletypeCompleter(
+    request_data )
+  LOGGER.debug( 'Using filetype completion: %s', do_filetype_completion )
+  filetypes = request_data[ 'filetypes' ]
+  completer = ( SERVER_STATE.GetFiletypeCompleter( filetypes ) if
+                do_filetype_completion else
+                SERVER_STATE.GetGeneralCompleter() )
+
+  return _JsonResponse( completer.ComputeCandidates( request_data ) )
+
+
+@app.get( '/user_options' )
+def GetUserOptions():
+  LOGGER.info( 'Received user options GET request' )
+  return _JsonResponse( dict( SERVER_STATE.user_options ) )
+
+
+@app.get( '/healthy' )
+def GetHealthy():
+  LOGGER.info( 'Received health request' )
+  return _JsonResponse( True )
+
+
+@app.post( '/user_options' )
+def SetUserOptions():
+  LOGGER.info( 'Received user options POST request' )
+  UpdateUserOptions( request.json )
+
+
+@app.post( '/semantic_completion_available' )
+def FiletypeCompletionAvailable():
+  LOGGER.info( 'Received filetype completion available request' )
+  return _JsonResponse( SERVER_STATE.FiletypeCompletionAvailable(
+      request.json[ 'filetypes' ] ) )
+
+
+@app.post( '/defined_subcommands' )
+def DefinedSubcommands():
+  LOGGER.info( 'Received defined subcommands request' )
+  completer = _GetCompleterForRequestData( request.json )
+
+  return _JsonResponse( completer.DefinedSubcommands() )
+
+
+@app.post( '/detailed_diagnostic' )
+def GetDetailedDiagnostic():
+  LOGGER.info( 'Received detailed diagnostic request' )
+  request_data = request.json
+  completer = _GetCompleterForRequestData( request_data )
+
+  return _JsonResponse( completer.GetDetailedDiagnostic( request_data ) )
+
+
+@app.post( '/load_extra_conf_file' )
+def LoadExtraConfFile():
+  LOGGER.info( 'Received extra conf load request' )
+  request_data = request.json
+  extra_conf_store.Load( request_data[ 'filepath' ], force = True )
+
+
+@app.post( '/debug_info' )
+def DebugInfo():
+  LOGGER.info( 'Received debug info request' )
+
+  output = []
+  has_clang_support = ycm_core.HasClangSupport()
+  output.append( 'Server has Clang support compiled in: {0}'.format(
+    has_clang_support ) )
+
+  if has_clang_support:
+    output.append( 'Clang version: ' + ycm_core.ClangVersion() )
+
+  request_data = request.json
+  try:
+    output.append(
+        _GetCompleterForRequestData( request_data ).DebugInfo( request_data) )
+  except:
+    pass
+  return _JsonResponse( '\n'.join( output ) )
+
+
+# The type of the param is Bottle.HTTPError
+@app.error( httplib.INTERNAL_SERVER_ERROR )
+def ErrorHandler( httperror ):
+  return _JsonResponse( BuildExceptionResponse( httperror.exception,
+                                                httperror.traceback ) )
+
+
+def _JsonResponse( data ):
+  response.set_header( 'Content-Type', 'application/json' )
+  return json.dumps( data, default = _UniversalSerialize )
+
+
+def _UniversalSerialize( obj ):
+  serialized = obj.__dict__.copy()
+  serialized[ 'TYPE' ] = type( obj ).__name__
+  return serialized
+
+
+def _GetCompleterForRequestData( request_data ):
+  completer_target = request_data.get( 'completer_target', None )
+
+  if completer_target == 'identifier':
+    return SERVER_STATE.GetGeneralCompleter().GetIdentifierCompleter()
+  elif completer_target == 'filetype_default' or not completer_target:
+    return SERVER_STATE.GetFiletypeCompleter( request_data[ 'filetypes' ] )
+  else:
+    return SERVER_STATE.GetFiletypeCompleter( [ completer_target ] )
+
+
+@atexit.register
+def ServerShutdown():
+  LOGGER.info( 'Server shutting down' )
+  if SERVER_STATE:
+    SERVER_STATE.Shutdown()
+    extra_conf_store.Shutdown()
+
+
+def UpdateUserOptions( options ):
+  global SERVER_STATE
+
+  if not options:
+    return
+
+  user_options_store.SetAll( options )
+  SERVER_STATE = server_state.ServerState( options )
+
+
+def SetServerStateToDefaults():
+  global SERVER_STATE, LOGGER
+  LOGGER = logging.getLogger( __name__ )
+  user_options_store.LoadDefaults()
+  SERVER_STATE = server_state.ServerState( user_options_store.GetAll() )
+  extra_conf_store.Reset()

python/ycm/server/responses.py

@@ -0,0 +1,104 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+CONFIRM_CONF_FILE_MESSAGE = ('Found {0}. Load? \n\n(Question can be turned '
+                             'off with options, see YCM docs)')
+
+class ServerError( Exception ):
+  def __init__( self, message ):
+    super( ServerError, self ).__init__( message )
+
+
+class UnknownExtraConf( ServerError ):
+  def __init__( self, extra_conf_file ):
+    message = CONFIRM_CONF_FILE_MESSAGE.format( extra_conf_file )
+    super( UnknownExtraConf, self ).__init__( message )
+    self.extra_conf_file = extra_conf_file
+
+
+
+def BuildGoToResponse( filepath, line_num, column_num, description = None ):
+  response = {
+    'filepath': os.path.realpath( filepath ),
+    'line_num': line_num,
+    'column_num': column_num
+  }
+
+  if description:
+    response[ 'description' ] = description
+  return response
+
+
+def BuildDescriptionOnlyGoToResponse( text ):
+  return {
+    'description': text,
+  }
+
+
+# TODO: Look at all the callers and ensure they are not using this instead of an
+# exception.
+def BuildDisplayMessageResponse( text ):
+  return {
+    'message': text
+  }
+
+
+def BuildCompletionData( insertion_text,
+                         extra_menu_info = None,
+                         detailed_info = None,
+                         menu_text = None,
+                         kind = None ):
+  completion_data = {
+    'insertion_text': insertion_text
+  }
+
+  if extra_menu_info:
+    completion_data[ 'extra_menu_info' ] = extra_menu_info
+  if menu_text:
+    completion_data[ 'menu_text' ] = menu_text
+  if detailed_info:
+    completion_data[ 'detailed_info' ] = detailed_info
+  if kind:
+    completion_data[ 'kind' ] = kind
+  return completion_data
+
+
+def BuildDiagnosticData( filepath,
+                         line_num,
+                         column_num,
+                         text,
+                         kind ):
+  return {
+    'filepath': filepath,
+    'line_num': line_num,
+    'column_num': column_num,
+    'text': text,
+    'kind': kind
+  }
+
+
+def BuildExceptionResponse( exception, traceback ):
+  return {
+    'exception': exception,
+    'message': str( exception ),
+    'traceback': traceback
+  }
+

python/ycm/server/server_state.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+import imp
+import os
+from ycm.utils import ForceSemanticCompletion
+from ycm.completers.general.general_completer_store import GeneralCompleterStore
+from ycm.completers.completer_utils import PathToFiletypeCompleterPluginLoader
+
+
+class ServerState( object ):
+  def __init__( self, user_options ):
+    self._user_options = user_options
+    self._filetype_completers = {}
+    self._gencomp = GeneralCompleterStore( self._user_options )
+
+
+  @property
+  def user_options( self ):
+    return self._user_options
+
+
+  def Shutdown( self ):
+    for completer in self._filetype_completers.itervalues():
+      if completer:
+        completer.Shutdown()
+
+    self._gencomp.Shutdown()
+
+
+  def _GetFiletypeCompleterForFiletype( self, filetype ):
+    try:
+      return self._filetype_completers[ filetype ]
+    except KeyError:
+      pass
+
+    module_path = PathToFiletypeCompleterPluginLoader( filetype )
+    completer = None
+    supported_filetypes = [ filetype ]
+    if os.path.exists( module_path ):
+      module = imp.load_source( filetype, module_path )
+      completer = module.GetCompleter( self._user_options )
+      if completer:
+        supported_filetypes.extend( completer.SupportedFiletypes() )
+
+    for supported_filetype in supported_filetypes:
+      self._filetype_completers[ supported_filetype ] = completer
+    return completer
+
+
+  def GetFiletypeCompleter( self, current_filetypes ):
+    completers = [ self._GetFiletypeCompleterForFiletype( filetype )
+                   for filetype in current_filetypes ]
+
+    for completer in completers:
+      if completer:
+        return completer
+
+    raise ValueError( 'No semantic completer exists for filetypes: {0}'.format(
+        current_filetypes ) )
+
+
+  def FiletypeCompletionAvailable( self, filetypes ):
+    try:
+      self.GetFiletypeCompleter( filetypes )
+      return True
+    except:
+      return False
+
+
+  def FiletypeCompletionUsable( self, filetypes ):
+    return ( self.CurrentFiletypeCompletionEnabled( filetypes ) and
+             self.FiletypeCompletionAvailable( filetypes ) )
+
+
+  def ShouldUseGeneralCompleter( self, request_data ):
+    return self._gencomp.ShouldUseNow( request_data )
+
+
+  def ShouldUseFiletypeCompleter( self, request_data ):
+    filetypes = request_data[ 'filetypes' ]
+    if self.FiletypeCompletionUsable( filetypes ):
+      return ( ForceSemanticCompletion( request_data ) or
+               self.GetFiletypeCompleter( filetypes ).ShouldUseNow(
+                 request_data ) )
+    return False
+
+
+  def GetGeneralCompleter( self ):
+    return self._gencomp
+
+
+  def CurrentFiletypeCompletionEnabled( self, current_filetypes ):
+    filetype_to_disable = self._user_options[
+        'filetype_specific_completion_to_disable' ]
+    return not all([ x in filetype_to_disable for x in current_filetypes ])
+

python/ycm/server/server_utils.py

@@ -0,0 +1,34 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+import os
+
+def SetUpPythonPath():
+  # We want to have the YouCompleteMe/python directory on the Python PATH
+  # because all the code already assumes that it's there. This is a relic from
+  # before the client/server architecture.
+  # TODO: Fix things so that this is not needed anymore when we split ycmd into
+  # a separate repository.
+  sys.path.insert( 0, os.path.join(
+                          os.path.dirname( os.path.abspath( __file__ ) ),
+                          '../..' ) )
+
+  from ycm import utils
+  utils.AddThirdPartyFoldersToSysPath()

python/ycm/server/tests/basic_test.py

@@ -0,0 +1,451 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import httplib
+import time
+from ..server_utils import SetUpPythonPath
+SetUpPythonPath()
+from webtest import TestApp
+from .. import handlers
+from ..responses import BuildCompletionData, UnknownExtraConf
+from nose.tools import ok_, eq_, with_setup
+from hamcrest import ( assert_that, has_items, has_entry, contains,
+                       contains_string, has_entries, contains_inanyorder )
+import bottle
+
+bottle.debug( True )
+
+# TODO: Split this file into multiple files.
+
+def BuildRequest( **kwargs ):
+  filepath = kwargs[ 'filepath' ] if 'filepath' in kwargs else '/foo'
+  contents = kwargs[ 'contents' ] if 'contents' in kwargs else ''
+  filetype = kwargs[ 'filetype' ] if 'filetype' in kwargs else 'foo'
+
+  request = {
+    'query': '',
+    'line_num': 0,
+    'column_num': 0,
+    'start_column': 0,
+    'filetypes': [ filetype ],
+    'filepath': filepath,
+    'line_value': contents,
+    'file_data': {
+      filepath: {
+        'contents': contents,
+        'filetypes': [ filetype ]
+      }
+    }
+  }
+
+  for key, value in kwargs.iteritems():
+    if key in [ 'contents', 'filetype', 'filepath' ]:
+      continue
+    request[ key ] = value
+
+    if key == 'line_num':
+      lines = contents.splitlines()
+      if len( lines ) > 1:
+        # NOTE: assumes 0-based line_num
+        request[ 'line_value' ] = lines[ value ]
+
+  return request
+
+
+# TODO: Make the other tests use this helper too instead of BuildCompletionData
+def CompletionEntryMatcher( insertion_text ):
+  return has_entry( 'insertion_text', insertion_text )
+
+
+def Setup():
+  handlers.SetServerStateToDefaults()
+
+
+def PathToTestDataDir():
+  dir_of_current_script = os.path.dirname( os.path.abspath( __file__ ) )
+  return os.path.join( dir_of_current_script, 'testdata' )
+
+
+def PathToTestFile( test_basename ):
+  return os.path.join( PathToTestDataDir(), test_basename )
+
+
+@with_setup( Setup )
+def GetCompletions_IdentifierCompleter_Works_test():
+  app = TestApp( handlers.app )
+  event_data = BuildRequest( contents = 'foo foogoo ba',
+                             event_name = 'FileReadyToParse' )
+
+  app.post_json( '/event_notification', event_data )
+
+  completion_data = BuildRequest( contents = 'oo foo foogoo ba',
+                                  query = 'oo',
+                                  column_num = 2 )
+
+  eq_( [ BuildCompletionData( 'foo' ),
+         BuildCompletionData( 'foogoo' ) ],
+       app.post_json( '/completions', completion_data ).json )
+
+
+@with_setup( Setup )
+def GetCompletions_CsCompleter_Works_test():
+  app = TestApp( handlers.app )
+  filepath = PathToTestFile( 'testy/Program.cs' )
+  contents = open( filepath ).read()
+  event_data = BuildRequest( filepath = filepath,
+                             filetype = 'cs',
+                             contents = contents,
+                             event_name = 'FileReadyToParse' )
+
+  app.post_json( '/event_notification', event_data )
+
+  # We need to wait until the server has started up.
+  while True:
+    result = app.post_json( '/run_completer_command',
+                            BuildRequest( completer_target = 'filetype_default',
+                                          command_arguments = ['ServerRunning'],
+                                          filetype = 'cs' ) ).json
+    if result:
+      break
+    time.sleep( 0.2 )
+
+  completion_data = BuildRequest( filepath = filepath,
+                                  filetype = 'cs',
+                                  contents = contents,
+                                  line_num = 8,
+                                  column_num = 11,
+                                  start_column = 11 )
+
+  results = app.post_json( '/completions', completion_data ).json
+  assert_that( results, has_items( CompletionEntryMatcher( 'CursorLeft' ),
+                                   CompletionEntryMatcher( 'CursorSize' ) ) )
+
+
+@with_setup( Setup )
+def GetCompletions_ClangCompleter_WorksWithExplicitFlags_test():
+  app = TestApp( handlers.app )
+  contents = """
+struct Foo {
+  int x;
+  int y;
+  char c;
+};
+
+int main()
+{
+  Foo foo;
+  foo.
+}
+"""
+
+  # 0-based line and column!
+  completion_data = BuildRequest( filepath = '/foo.cpp',
+                                  filetype = 'cpp',
+                                  contents = contents,
+                                  line_num = 10,
+                                  column_num = 6,
+                                  start_column = 6,
+                                  compilation_flags = ['-x', 'c++'] )
+
+  results = app.post_json( '/completions', completion_data ).json
+  assert_that( results, has_items( CompletionEntryMatcher( 'c' ),
+                                   CompletionEntryMatcher( 'x' ),
+                                   CompletionEntryMatcher( 'y' ) ) )
+
+
+@with_setup( Setup )
+def GetCompletions_ClangCompleter_UnknownExtraConfException_test():
+  app = TestApp( handlers.app )
+  filepath = PathToTestFile( 'basic.cpp' )
+  completion_data = BuildRequest( filepath = filepath,
+                                  filetype = 'cpp',
+                                  contents = open( filepath ).read(),
+                                  force_semantic = True )
+
+  response = app.post_json( '/completions',
+                            completion_data,
+                            expect_errors = True )
+
+  eq_( response.status_code, httplib.INTERNAL_SERVER_ERROR )
+  assert_that( response.json,
+               has_entry( 'exception',
+                          has_entry( 'TYPE', UnknownExtraConf.__name__ ) ) )
+
+
+@with_setup( Setup )
+def GetCompletions_ClangCompleter_WorksWhenExtraConfExplicitlyAllowed_test():
+  app = TestApp( handlers.app )
+  app.post_json( '/load_extra_conf_file',
+                 { 'filepath': PathToTestFile( '.ycm_extra_conf.py' ) } )
+
+  filepath = PathToTestFile( 'basic.cpp' )
+  completion_data = BuildRequest( filepath = filepath,
+                                  filetype = 'cpp',
+                                  contents = open( filepath ).read(),
+                                  line_num = 10,
+                                  column_num = 6,
+                                  start_column = 6 )
+
+  results = app.post_json( '/completions', completion_data ).json
+  assert_that( results, has_items( CompletionEntryMatcher( 'c' ),
+                                   CompletionEntryMatcher( 'x' ),
+                                   CompletionEntryMatcher( 'y' ) ) )
+
+
+@with_setup( Setup )
+def GetCompletions_ClangCompleter_ForceSemantic_OnlyFileteredCompletions_test():
+  app = TestApp( handlers.app )
+  contents = """
+int main()
+{
+  int foobar;
+  int floozar;
+  int gooboo;
+  int bleble;
+
+  fooar
+}
+"""
+
+  # 0-based line and column!
+  completion_data = BuildRequest( filepath = '/foo.cpp',
+                                  filetype = 'cpp',
+                                  force_semantic = True,
+                                  contents = contents,
+                                  line_num = 8,
+                                  column_num = 7,
+                                  start_column = 7,
+                                  query = 'fooar',
+                                  compilation_flags = ['-x', 'c++'] )
+
+  results = app.post_json( '/completions', completion_data ).json
+  assert_that( results,
+               contains_inanyorder( CompletionEntryMatcher( 'foobar' ),
+                                    CompletionEntryMatcher( 'floozar' ) ) )
+
+
+@with_setup( Setup )
+def GetCompletions_ForceSemantic_Works_test():
+  app = TestApp( handlers.app )
+
+  completion_data = BuildRequest( filetype = 'python',
+                                  force_semantic = True )
+
+  results = app.post_json( '/completions', completion_data ).json
+  assert_that( results, has_items( CompletionEntryMatcher( 'abs' ),
+                                   CompletionEntryMatcher( 'open' ),
+                                   CompletionEntryMatcher( 'bool' ) ) )
+
+
+@with_setup( Setup )
+def GetCompletions_IdentifierCompleter_SyntaxKeywordsAdded_test():
+  app = TestApp( handlers.app )
+  event_data = BuildRequest( event_name = 'FileReadyToParse',
+                             syntax_keywords = ['foo', 'bar', 'zoo'] )
+
+  app.post_json( '/event_notification', event_data )
+
+  completion_data = BuildRequest( contents =  'oo ',
+                                  query = 'oo',
+                                  column_num = 2 )
+
+  eq_( [ BuildCompletionData( 'foo' ),
+         BuildCompletionData( 'zoo' ) ],
+       app.post_json( '/completions', completion_data ).json )
+
+
+@with_setup( Setup )
+def GetCompletions_UltiSnipsCompleter_Works_test():
+  app = TestApp( handlers.app )
+  event_data = BuildRequest(
+    event_name = 'BufferVisit',
+    ultisnips_snippets = [
+        {'trigger': 'foo', 'description': 'bar'},
+        {'trigger': 'zoo', 'description': 'goo'},
+    ] )
+
+  app.post_json( '/event_notification', event_data )
+
+  completion_data = BuildRequest( contents =  'oo ',
+                                  query = 'oo',
+                                  column_num = 2 )
+
+  eq_( [ BuildCompletionData( 'foo', '<snip> bar' ),
+         BuildCompletionData( 'zoo', '<snip> goo' ) ],
+       app.post_json( '/completions', completion_data ).json )
+
+
+@with_setup( Setup )
+def RunCompleterCommand_GoTo_Jedi_ZeroBasedLineAndColumn_test():
+  app = TestApp( handlers.app )
+  contents = """
+def foo():
+  pass
+
+foo()
+"""
+
+  goto_data = BuildRequest( completer_target = 'filetype_default',
+                            command_arguments = ['GoToDefinition'],
+                            line_num = 4,
+                            contents = contents,
+                            filetype = 'python',
+                            filepath = '/foo.py' )
+
+  # 0-based line and column!
+  eq_( {
+         'filepath': '/foo.py',
+         'line_num': 1,
+         'column_num': 4
+       },
+       app.post_json( '/run_completer_command', goto_data ).json )
+
+
+@with_setup( Setup )
+def RunCompleterCommand_GoTo_Clang_ZeroBasedLineAndColumn_test():
+  app = TestApp( handlers.app )
+  contents = """
+struct Foo {
+  int x;
+  int y;
+  char c;
+};
+
+int main()
+{
+  Foo foo;
+  return 0;
+}
+"""
+
+  goto_data = BuildRequest( completer_target = 'filetype_default',
+                            command_arguments = ['GoToDefinition'],
+                            compilation_flags = ['-x', 'c++'],
+                            line_num = 9,
+                            column_num = 2,
+                            contents = contents,
+                            filetype = 'cpp' )
+
+  # 0-based line and column!
+  eq_( {
+        'filepath': '/foo',
+        'line_num': 1,
+        'column_num': 7
+      },
+      app.post_json( '/run_completer_command', goto_data ).json )
+
+
+@with_setup( Setup )
+def DefinedSubcommands_Works_test():
+  app = TestApp( handlers.app )
+  subcommands_data = BuildRequest( completer_target = 'python' )
+
+  eq_( [ 'GoToDefinition',
+         'GoToDeclaration',
+         'GoToDefinitionElseDeclaration' ],
+       app.post_json( '/defined_subcommands', subcommands_data ).json )
+
+
+@with_setup( Setup )
+def DefinedSubcommands_WorksWhenNoExplicitCompleterTargetSpecified_test():
+  app = TestApp( handlers.app )
+  subcommands_data = BuildRequest( filetype = 'python' )
+
+  eq_( [ 'GoToDefinition',
+         'GoToDeclaration',
+         'GoToDefinitionElseDeclaration' ],
+       app.post_json( '/defined_subcommands', subcommands_data ).json )
+
+
+@with_setup( Setup )
+def Diagnostics_ClangCompleter_ZeroBasedLineAndColumn_test():
+  app = TestApp( handlers.app )
+  contents = """
+struct Foo {
+  int x  // semicolon missing here!
+  int y;
+  int c;
+  int d;
+};
+"""
+
+  event_data = BuildRequest( compilation_flags = ['-x', 'c++'],
+                             event_name = 'FileReadyToParse',
+                             contents = contents,
+                             filetype = 'cpp' )
+
+  results = app.post_json( '/event_notification', event_data ).json
+  assert_that( results,
+               contains(
+                  has_entries( { 'text': contains_string( "expected ';'" ),
+                                 'line_num': 2,
+                                 'column_num': 7 } ) ) )
+
+
+@with_setup( Setup )
+def GetDetailedDiagnostic_ClangCompleter_Works_test():
+  app = TestApp( handlers.app )
+  contents = """
+struct Foo {
+  int x  // semicolon missing here!
+  int y;
+  int c;
+  int d;
+};
+"""
+
+  diag_data = BuildRequest( compilation_flags = ['-x', 'c++'],
+                            line_num = 2,
+                            contents = contents,
+                            filetype = 'cpp' )
+
+  event_data = diag_data.copy()
+  event_data.update( {
+    'event_name': 'FileReadyToParse',
+  } )
+
+  app.post_json( '/event_notification', event_data )
+  results = app.post_json( '/detailed_diagnostic', diag_data ).json
+  assert_that( results,
+               has_entry( 'message', contains_string( "expected ';'" ) ) )
+
+
+@with_setup( Setup )
+def FiletypeCompletionAvailable_Works_test():
+  app = TestApp( handlers.app )
+  request_data = {
+    'filetypes': ['python']
+  }
+
+  ok_( app.post_json( '/semantic_completion_available',
+                      request_data ).json )
+
+
+@with_setup( Setup )
+def UserOptions_Works_test():
+  app = TestApp( handlers.app )
+  options = app.get( '/user_options' ).json
+  ok_( len( options ) )
+
+  options[ 'foobar' ] = 'zoo'
+
+  app.post_json( '/user_options', options )
+  eq_( options, app.get( '/user_options' ).json )
+

python/ycm/server/tests/testdata/.ycm_extra_conf.py

@@ -0,0 +1,5 @@
+def FlagsForFile( filename ):
+  return {
+    'flags': ['-x', 'c++'],
+    'do_cache': True
+  }

python/ycm/server/tests/testdata/basic.cpp

@@ -0,0 +1,13 @@
+struct Foo {
+  int x;
+  int y;
+  char c;
+};
+
+int main()
+{
+  Foo foo;
+  // The location after the dot is line 11, col 7
+  foo.
+}
+

python/ycm/server/tests/testdata/testy/Program.cs

@@ -0,0 +1,12 @@
+using System;
+
+namespace testy
+{
+	class MainClass
+	{
+		public static void Main (string[] args)
+		{
+			Console.
+		}
+	}
+}

python/ycm/server/tests/testdata/testy/Properties/AssemblyInfo.cs

@@ -0,0 +1,22 @@
+using System.Reflection;
+using System.Runtime.CompilerServices;
+
+// Information about this assembly is defined by the following attributes. 
+// Change them to the values specific to your project.
+[assembly: AssemblyTitle ("testy")]
+[assembly: AssemblyDescription ("")]
+[assembly: AssemblyConfiguration ("")]
+[assembly: AssemblyCompany ("")]
+[assembly: AssemblyProduct ("")]
+[assembly: AssemblyCopyright ("valloric")]
+[assembly: AssemblyTrademark ("")]
+[assembly: AssemblyCulture ("")]
+// The assembly version has the format "{Major}.{Minor}.{Build}.{Revision}".
+// The form "{Major}.{Minor}.*" will automatically update the build and revision,
+// and "{Major}.{Minor}.{Build}.*" will update just the revision.
+[assembly: AssemblyVersion ("1.0.*")]
+// The following attributes are used to specify the signing key for the assembly, 
+// if desired. See the Mono documentation for more information about signing.
+//[assembly: AssemblyDelaySign(false)]
+//[assembly: AssemblyKeyFile("")]
+

python/ycm/server/tests/testdata/testy/testy.csproj

@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <PropertyGroup>
+    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
+    <Platform Condition=" '$(Platform)' == '' ">x86</Platform>
+    <ProductVersion>10.0.0</ProductVersion>
+    <SchemaVersion>2.0</SchemaVersion>
+    <ProjectGuid>{0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}</ProjectGuid>
+    <OutputType>Exe</OutputType>
+    <RootNamespace>testy</RootNamespace>
+    <AssemblyName>testy</AssemblyName>
+  </PropertyGroup>
+  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|x86' ">
+    <DebugSymbols>true</DebugSymbols>
+    <DebugType>full</DebugType>
+    <Optimize>false</Optimize>
+    <OutputPath>bin\Debug</OutputPath>
+    <DefineConstants>DEBUG;</DefineConstants>
+    <ErrorReport>prompt</ErrorReport>
+    <WarningLevel>4</WarningLevel>
+    <Externalconsole>true</Externalconsole>
+    <PlatformTarget>x86</PlatformTarget>
+  </PropertyGroup>
+  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|x86' ">
+    <DebugType>full</DebugType>
+    <Optimize>true</Optimize>
+    <OutputPath>bin\Release</OutputPath>
+    <ErrorReport>prompt</ErrorReport>
+    <WarningLevel>4</WarningLevel>
+    <Externalconsole>true</Externalconsole>
+    <PlatformTarget>x86</PlatformTarget>
+  </PropertyGroup>
+  <ItemGroup>
+    <Reference Include="System" />
+  </ItemGroup>
+  <ItemGroup>
+    <Compile Include="Program.cs" />
+    <Compile Include="Properties\AssemblyInfo.cs" />
+  </ItemGroup>
+  <Import Project="$(MSBuildBinPath)\Microsoft.CSharp.targets" />
+</Project>

python/ycm/server/tests/testdata/testy/testy.sln

@@ -0,0 +1,20 @@
+
+Microsoft Visual Studio Solution File, Format Version 11.00
+# Visual Studio 2010
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "testy", "testy.csproj", "{0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}"
+EndProject
+Global
+	GlobalSection(SolutionConfigurationPlatforms) = preSolution
+		Debug|x86 = Debug|x86
+		Release|x86 = Release|x86
+	EndGlobalSection
+	GlobalSection(ProjectConfigurationPlatforms) = postSolution
+		{0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}.Debug|x86.ActiveCfg = Debug|x86
+		{0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}.Debug|x86.Build.0 = Debug|x86
+		{0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}.Release|x86.ActiveCfg = Release|x86
+		{0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}.Release|x86.Build.0 = Release|x86
+	EndGlobalSection
+	GlobalSection(MonoDevelopProperties) = preSolution
+		StartupItem = testy.csproj
+	EndGlobalSection
+EndGlobal

python/ycm/server/tests/testdata/testy/testy.userprefs

@@ -0,0 +1,13 @@
+<Properties>
+  <MonoDevelop.Ide.Workspace ActiveConfiguration="Debug|x86" />
+  <MonoDevelop.Ide.Workbench ActiveDocument="Program.cs">
+    <Files>
+      <File FileName="Program.cs" Line="7" Column="25" />
+      <File FileName="Properties/AssemblyInfo.cs" Line="1" Column="1" />
+    </Files>
+  </MonoDevelop.Ide.Workbench>
+  <MonoDevelop.Ide.DebuggingService.Breakpoints>
+    <BreakpointStore />
+  </MonoDevelop.Ide.DebuggingService.Breakpoints>
+  <MonoDevelop.Ide.DebuggingService.PinnedWatches />
+</Properties>

python/ycm/server/watchdog_plugin.py

@@ -0,0 +1,78 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+import time
+import os
+import copy
+from ycm import utils
+from threading import Thread, Lock
+
+# This class implements the Bottle plugin API:
+# http://bottlepy.org/docs/dev/plugindev.html
+#
+# The idea here is to decorate every route handler automatically so that on
+# every request, we log when the request was made. Then a watchdog thread checks
+# every check_interval_seconds whether the server has been idle for a time
+# greater that the passed-in idle_suicide_seconds. If it has, we kill the
+# server.
+#
+# We want to do this so that if something goes bonkers in Vim and the server
+# never gets killed by the client, we don't end up with lots of zombie servers.
+class WatchdogPlugin( object ):
+  name = 'watchdog'
+  api = 2
+
+
+  def __init__( self,
+                idle_suicide_seconds,
+                check_interval_seconds = 60 * 10 ):
+    self._check_interval_seconds = check_interval_seconds
+    self._idle_suicide_seconds = idle_suicide_seconds
+    self._last_request_time = time.time()
+    self._last_request_time_lock = Lock()
+    if idle_suicide_seconds <= 0:
+      return
+    self._watchdog_thread = Thread( target = self._WatchdogMain )
+    self._watchdog_thread.daemon = True
+    self._watchdog_thread.start()
+
+
+  def _GetLastRequestTime( self ):
+    with self._last_request_time_lock:
+      return copy.deepcopy( self._last_request_time )
+
+
+  def _SetLastRequestTime( self, new_value ):
+    with self._last_request_time_lock:
+      self._last_request_time = new_value
+
+
+  def _WatchdogMain( self ):
+    while True:
+      time.sleep( self._check_interval_seconds )
+      if time.time() - self._GetLastRequestTime() > self._idle_suicide_seconds:
+        utils.TerminateProcess( os.getpid() )
+
+
+  def __call__( self, callback ):
+    def wrapper( *args, **kwargs ):
+      self._SetLastRequestTime( time.time() )
+      return callback( *args, **kwargs )
+    return wrapper
+

python/ycm/server/ycmd.py

@@ -0,0 +1,97 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+from server_utils import SetUpPythonPath
+SetUpPythonPath()
+
+import sys
+import logging
+import json
+import argparse
+import waitress
+import signal
+from ycm import user_options_store
+from ycm import extra_conf_store
+from ycm.server.watchdog_plugin import WatchdogPlugin
+
+
+def YcmCoreSanityCheck():
+  if 'ycm_core' in sys.modules:
+    raise RuntimeError( 'ycm_core already imported, ycmd has a bug!' )
+
+
+# We manually call sys.exit() on SIGTERM and SIGINT so that atexit handlers are
+# properly executed.
+def SetUpSignalHandler():
+  def SignalHandler( signum, frame ):
+    sys.exit()
+
+  for sig in [ signal.SIGTERM,
+               signal.SIGINT ]:
+    signal.signal( sig, SignalHandler )
+
+
+def Main():
+  parser = argparse.ArgumentParser()
+  parser.add_argument( '--host', type = str, default = 'localhost',
+                       help = 'server hostname')
+  parser.add_argument( '--port', type = int, default = 6666,
+                       help = 'server port')
+  parser.add_argument( '--log', type = str, default = 'info',
+                       help = 'log level, one of '
+                              '[debug|info|warning|error|critical]' )
+  parser.add_argument( '--idle_suicide_seconds', type = int, default = 0,
+                       help = 'num idle seconds before server shuts down')
+  parser.add_argument( '--options_file', type = str, default = '',
+                       help = 'file with user options, in JSON format' )
+  args = parser.parse_args()
+
+  numeric_level = getattr( logging, args.log.upper(), None )
+  if not isinstance( numeric_level, int ):
+    raise ValueError( 'Invalid log level: %s' % args.log )
+
+  # Has to be called before any call to logging.getLogger()
+  logging.basicConfig( format = '%(asctime)s - %(levelname)s - %(message)s',
+                       level = numeric_level )
+
+  options = None
+  if args.options_file:
+    options = json.load( open( args.options_file, 'r' ) )
+    user_options_store.SetAll( options )
+    # This ensures that ycm_core is not loaded before extra conf
+    # preload was run.
+    YcmCoreSanityCheck()
+    extra_conf_store.CallGlobalExtraConfYcmCorePreloadIfExists()
+
+  # This can't be a top-level import because it transitively imports
+  # ycm_core which we want to be imported ONLY after extra conf
+  # preload has executed.
+  from ycm.server import handlers
+  handlers.UpdateUserOptions( options )
+  SetUpSignalHandler()
+  handlers.app.install( WatchdogPlugin( args.idle_suicide_seconds ) )
+  waitress.serve( handlers.app,
+                  host = args.host,
+                  port = args.port,
+                  threads = 10 )
+
+
+if __name__ == "__main__":
+  Main()
+

python/ycm/user_options_store.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2013  Strahinja Val Markovic  <val@markovic.io>
+#
+# This file is part of YouCompleteMe.
+#
+# YouCompleteMe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# YouCompleteMe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+import os
+from frozendict import frozendict
+
+_USER_OPTIONS = {}
+
+def SetAll( new_options ):
+  global _USER_OPTIONS
+  _USER_OPTIONS = frozendict( new_options )
+
+
+def GetAll():
+  return _USER_OPTIONS
+
+
+def Value( key ):
+  return _USER_OPTIONS[ key ]
+
+
+def LoadDefaults():
+  SetAll( DefaultOptions() )
+
+
+def DefaultOptions():
+  settings_path = os.path.join(
+      os.path.dirname( os.path.abspath( __file__ ) ),
+      'server/default_settings.json' )
+  with open( settings_path ) as f:
+    return json.loads( f.read() )
+

python/ycm/utils.py

@@ -17,6 +17,18 @@
 # You should have received a copy of the GNU General Public License
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
+import tempfile
+import os
+import sys
+import signal
+import functools
+import socket
+import stat
+
+WIN_PYTHON27_PATH = 'C:\python27\pythonw.exe'
+WIN_PYTHON26_PATH = 'C:\python26\pythonw.exe'
+
+
 def IsIdentifierChar( char ):
   return char.isalnum() or char == '_'
 
@@ -24,3 +36,99 @@ def IsIdentifierChar( char ):
 def SanitizeQuery( query ):
   return query.strip()
 
+
+def ToUtf8IfNeeded( string_or_unicode ):
+  if isinstance( string_or_unicode, unicode ):
+    return string_or_unicode.encode( 'utf8' )
+  return string_or_unicode
+
+
+def PathToTempDir():
+  tempdir = os.path.join( tempfile.gettempdir(), 'ycm_temp' )
+  if not os.path.exists( tempdir ):
+    os.makedirs( tempdir )
+    # Needed to support multiple users working on the same machine;
+    # see issue 606.
+    MakeFolderAccessibleToAll( tempdir )
+  return tempdir
+
+
+def MakeFolderAccessibleToAll( path_to_folder ):
+  current_stat = os.stat( path_to_folder )
+  # readable, writable and executable by everyone
+  flags = ( current_stat.st_mode | stat.S_IROTH | stat.S_IWOTH | stat.S_IXOTH
+            | stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP )
+  os.chmod( path_to_folder, flags )
+
+
+def GetUnusedLocalhostPort():
+  sock = socket.socket()
+  # This tells the OS to give us any free port in the range [1024 - 65535]
+  sock.bind( ( '', 0 ) )
+  port = sock.getsockname()[ 1 ]
+  sock.close()
+  return port
+
+
+def PathToPythonInterpreter():
+  # This is a bit tricky. Normally, sys.executable has the full path to the
+  # Python interpreter. But this code is also executed from inside Vim's
+  # embedded Python. On Unix machines, even that Python returns a good value for
+  # sys.executable, but on Windows it returns the path to the Vim binary, which
+  # is useless to us (issue #581). So we check the common install location for
+  # Python on Windows, first for Python 2.7 and then for 2.6.
+  #
+  # I'm open to better ideas on how to do this.
+
+  if OnWindows():
+    if os.path.exists( WIN_PYTHON27_PATH ):
+      return WIN_PYTHON27_PATH
+    elif os.path.exists( WIN_PYTHON26_PATH ):
+      return WIN_PYTHON26_PATH
+    raise RuntimeError( 'Python 2.7/2.6 not installed!' )
+  else:
+    return sys.executable
+
+
+def OnWindows():
+  return sys.platform == 'win32'
+
+
+# From here: http://stackoverflow.com/a/8536476/1672783
+def TerminateProcess( pid ):
+  if OnWindows():
+    import ctypes
+    PROCESS_TERMINATE = 1
+    handle = ctypes.windll.kernel32.OpenProcess( PROCESS_TERMINATE,
+                                                 False,
+                                                 pid )
+    ctypes.windll.kernel32.TerminateProcess( handle, -1 )
+    ctypes.windll.kernel32.CloseHandle( handle )
+  else:
+    os.kill( pid, signal.SIGTERM )
+
+
+def AddThirdPartyFoldersToSysPath():
+  path_to_third_party = os.path.join(
+                          os.path.dirname( os.path.abspath( __file__ ) ),
+                          '../../third_party' )
+
+  for folder in os.listdir( path_to_third_party ):
+    sys.path.insert( 0, os.path.realpath( os.path.join( path_to_third_party,
+                                                        folder ) ) )
+
+def Memoize( obj ):
+  cache = obj.cache = {}
+
+  @functools.wraps( obj )
+  def memoizer( *args, **kwargs ):
+    key = str( args ) + str( kwargs )
+    if key not in cache:
+      cache[ key ] = obj( *args, **kwargs )
+    return cache[ key ]
+  return memoizer
+
+
+def ForceSemanticCompletion( request_data ):
+  return ( 'force_semantic' in request_data and
+           bool( request_data[ 'force_semantic' ] ) )

python/ycm/vimsupport.py

@@ -18,6 +18,8 @@
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
 import vim
+import os
+import json
 
 def CurrentLineAndColumn():
   """Returns the 0-based current line and 0-based current column."""
@@ -46,12 +48,74 @@ def TextAfterCursor():
   return vim.current.line[ CurrentColumn(): ]
 
 
-def GetUnsavedBuffers():
+# Note the difference between buffer OPTIONS and VARIABLES; the two are not
-  def BufferModified( buffer_number ):
+# the same.
-    to_eval = 'getbufvar({0}, "&mod")'.format( buffer_number )
+def GetBufferOption( buffer_object, option ):
-    return GetBoolValue( to_eval )
+  # The 'options' property is only available in recent (7.4+) Vim builds
+  if hasattr( buffer_object, 'options' ):
+    return buffer_object.options[ option ]
 
-  return ( x for x in vim.buffers if BufferModified( x.number ) )
+  to_eval = 'getbufvar({0}, "&{1}")'.format( buffer_object.number, option )
+  return GetVariableValue( to_eval )
+
+
+def GetUnsavedAndCurrentBufferData():
+  def BufferModified( buffer_object ):
+    return bool( int( GetBufferOption( buffer_object, 'mod' ) ) )
+
+  buffers_data = {}
+  for buffer_object in vim.buffers:
+    if not ( BufferModified( buffer_object ) or
+             buffer_object == vim.current.buffer ):
+      continue
+
+    buffers_data[ GetBufferFilepath( buffer_object ) ] = {
+      'contents': '\n'.join( buffer_object ),
+      'filetypes': FiletypesForBuffer( buffer_object )
+    }
+
+  return buffers_data
+
+
+def GetBufferNumberForFilename( filename, open_file_if_needed = True ):
+  return int( vim.eval( "bufnr('{0}', {1})".format(
+      os.path.realpath( filename ),
+      int( open_file_if_needed ) ) ) )
+
+
+def GetCurrentBufferFilepath():
+  return GetBufferFilepath( vim.current.buffer )
+
+
+def GetBufferFilepath( buffer_object ):
+  if buffer_object.name:
+    return buffer_object.name
+  # Buffers that have just been created by a command like :enew don't have any
+  # buffer name so we use the buffer number for that.
+  return os.path.join( os.getcwd(), str( buffer_object.number ) )
+
+
+# Given a dict like {'a': 1}, loads it into Vim as if you ran 'let g:a = 1'
+# When |overwrite| is True, overwrites the existing value in Vim.
+def LoadDictIntoVimGlobals( new_globals, overwrite = True ):
+  extend_option = '"force"' if overwrite else '"keep"'
+
+  # We need to use json.dumps because that won't use the 'u' prefix on strings
+  # which Vim would bork on.
+  vim.eval( 'extend( g:, {0}, {1})'.format( json.dumps( new_globals ),
+                                            extend_option ) )
+
+
+# Changing the returned dict will NOT change the value in Vim.
+def GetReadOnlyVimGlobals( force_python_objects = False ):
+  if force_python_objects:
+    return vim.eval( 'g:' )
+
+  try:
+    # vim.vars is fairly new so it might not exist
+    return vim.vars
+  except:
+    return vim.eval( 'g:' )
 
 
 # Both |line| and |column| need to be 1-based
@@ -59,7 +123,7 @@ def JumpToLocation( filename, line, column ):
   # Add an entry to the jumplist
   vim.command( "normal! m'" )
 
-  if filename != vim.current.buffer.name:
+  if filename != GetCurrentBufferFilepath():
     # We prefix the command with 'keepjumps' so that opening the file is not
     # recorded in the jumplist. So when we open the file and move the cursor to
     # a location in it, the user can use CTRL-O to jump back to the original
@@ -73,18 +137,24 @@ def JumpToLocation( filename, line, column ):
   vim.command( 'normal! zz' )
 
 
-def NumLinesInBuffer( buffer ):
+def NumLinesInBuffer( buffer_object ):
   # This is actually less than obvious, that's why it's wrapped in a function
-  return len( buffer )
+  return len( buffer_object )
 
 
+# Calling this function from the non-GUI thread will sometimes crash Vim. At the
+# time of writing, YCM only uses the GUI thread inside Vim (this used to not be
+# the case).
 def PostVimMessage( message ):
-  # TODO: Check are we on the main thread or not, and if not, force a crash
+  vim.command( "echohl WarningMsg | echom '{0}' | echohl None"
-  # here. This should make it impossible to accidentally call this from a
+               .format( EscapeForVim( str( message ) ) ) )
-  # non-GUI thread which *sometimes* crashes Vim because Vim is not thread-safe.
+
-  # A consistent crash should force us to notice the error.
+# Unlike PostVimMesasge, this supports messages with newlines in them because it
-  vim.command( "echohl WarningMsg | echomsg '{0}' | echohl None"
+# uses 'echo' instead of 'echomsg'. This also means that the message will NOT
-               .format( EscapeForVim( message ) ) )
+# appear in Vim's message log.
+def PostMultiLineNotice( message ):
+  vim.command( "echohl WarningMsg | echo '{0}' | echohl None"
+               .format( EscapeForVim( str( message ) ) ) )
 
 
 def PresentDialog( message, choices, default_choice_index = 0 ):
@@ -128,15 +198,13 @@ def EscapeForVim( text ):
 
 
 def CurrentFiletypes():
-  ft_string = vim.eval( "&filetype" )
+  return vim.eval( "&filetype" ).split( '.' )
-  return ft_string.split( '.' )
 
 
 def FiletypesForBuffer( buffer_object ):
   # NOTE: Getting &ft for other buffers only works when the buffer has been
   # visited by the user at least once, which is true for modified buffers
-  ft_string = vim.eval( 'getbufvar({0}, "&ft")'.format( buffer_object.number ) )
+  return GetBufferOption( buffer_object, 'ft' ).split( '.' )
-  return ft_string.split( '.' )
 
 
 def GetVariableValue( variable ):
@@ -145,3 +213,8 @@ def GetVariableValue( variable ):
 
 def GetBoolValue( variable ):
   return bool( int( vim.eval( variable ) ) )
+
+
+def GetIntValue( variable ):
+  return int( vim.eval( variable ) )
+

python/ycm/youcompleteme.py

@@ -17,203 +17,301 @@
 # You should have received a copy of the GNU General Public License
 # along with YouCompleteMe.  If not, see <http://www.gnu.org/licenses/>.
 
-import imp
 import os
 import vim
-import ycm_core
+import subprocess
+import tempfile
+import json
 from ycm import vimsupport
+from ycm import utils
 from ycm.completers.all.omni_completer import OmniCompleter
-from ycm.completers.general.general_completer_store import GeneralCompleterStore
+from ycm.completers.general import syntax_parse
+from ycm.completers.completer_utils import FiletypeCompleterExistsForFiletype
+from ycm.client.base_request import BaseRequest, BuildRequestData
+from ycm.client.command_request import SendCommandRequest
+from ycm.client.completion_request import CompletionRequest
+from ycm.client.omni_completion_request import OmniCompletionRequest
+from ycm.client.event_notification import ( SendEventNotificationAsync,
+                                            EventNotification )
+from ycm.server.responses import ServerError
 
+try:
+  from UltiSnips import UltiSnips_Manager
+  USE_ULTISNIPS_DATA = True
+except ImportError:
+  USE_ULTISNIPS_DATA = False
 
-FILETYPE_SPECIFIC_COMPLETION_TO_DISABLE = vim.eval(
+SERVER_CRASH_MESSAGE_STDERR_FILE = 'The ycmd server SHUT DOWN with output:\n'
-  'g:ycm_filetype_specific_completion_to_disable' )
+SERVER_CRASH_MESSAGE_SAME_STDERR = (
+  'The ycmd server shut down, check console output for logs!' )
 
 
 class YouCompleteMe( object ):
-  def __init__( self ):
+  def __init__( self, user_options ):
-    self.gencomp = GeneralCompleterStore()
+    self._user_options = user_options
-    self.omnicomp = OmniCompleter()
+    self._omnicomp = OmniCompleter( user_options )
-    self.filetype_completers = {}
+    self._latest_completion_request = None
+    self._latest_file_parse_request = None
+    self._server_stdout = None
+    self._server_stderr = None
+    self._server_popen = None
+    self._filetypes_with_keywords_loaded = set()
+    self._temp_options_filename = None
+    self._SetupServer()
 
 
-  def GetGeneralCompleter( self ):
+  def _SetupServer( self ):
-    return self.gencomp
+    server_port = utils.GetUnusedLocalhostPort()
+    with tempfile.NamedTemporaryFile( delete = False ) as options_file:
+      self._temp_options_filename = options_file.name
+      json.dump( dict( self._user_options ), options_file )
+      args = [ utils.PathToPythonInterpreter(),
+               _PathToServerScript(),
+               '--port={0}'.format( server_port ),
+               '--options_file={0}'.format( options_file.name ),
+               '--log={0}'.format( self._user_options[ 'server_log_level' ] ),
+               '--idle_suicide_seconds={0}'.format(
+                  self._user_options[ 'server_idle_suicide_seconds' ] ) ]
+
+      BaseRequest.server_location = 'http://localhost:' + str( server_port )
+
+      if self._user_options[ 'server_use_vim_stdout' ]:
+        self._server_popen = subprocess.Popen( args )
+      else:
+        filename_format = os.path.join( utils.PathToTempDir(),
+                                        'server_{port}_{std}.log' )
+
+        self._server_stdout = filename_format.format( port = server_port,
+                                                      std = 'stdout' )
+        self._server_stderr = filename_format.format( port = server_port,
+                                                      std = 'stderr' )
+
+        with open( self._server_stderr, 'w' ) as fstderr:
+          with open( self._server_stdout, 'w' ) as fstdout:
+            self._server_popen = subprocess.Popen( args,
+                                                   stdout = fstdout,
+                                                   stderr = fstderr )
+    self._NotifyUserIfServerCrashed()
+
+
+  def _IsServerAlive( self ):
+    returncode = self._server_popen.poll()
+    # When the process hasn't finished yet, poll() returns None.
+    return returncode is None
+
+
+  def _NotifyUserIfServerCrashed( self ):
+    if self._IsServerAlive():
+      return
+    if self._server_stderr:
+      with open( self._server_stderr, 'r' ) as server_stderr_file:
+        vimsupport.PostMultiLineNotice( SERVER_CRASH_MESSAGE_STDERR_FILE +
+                                        server_stderr_file.read() )
+    else:
+        vimsupport.PostVimMessage( SERVER_CRASH_MESSAGE_SAME_STDERR )
+
+
+  def ServerPid( self ):
+    if not self._server_popen:
+      return -1
+    return self._server_popen.pid
+
+
+  def RestartServer( self ):
+    vimsupport.PostVimMessage( 'Restarting ycmd server...' )
+    self.OnVimLeave()
+    self._SetupServer()
+
+
+  def CreateCompletionRequest( self, force_semantic = False ):
+    # We have to store a reference to the newly created CompletionRequest
+    # because VimScript can't store a reference to a Python object across
+    # function calls... Thus we need to keep this request somewhere.
+    if ( not self.NativeFiletypeCompletionAvailable() and
+         self.CurrentFiletypeCompletionEnabled() and
+         self._omnicomp.ShouldUseNow() ):
+      self._latest_completion_request = OmniCompletionRequest( self._omnicomp )
+    else:
+      self._latest_completion_request = ( CompletionRequest( force_semantic )
+                                          if self._IsServerAlive() else
+                                          None )
+    return self._latest_completion_request
+
+
+  def SendCommandRequest( self, arguments, completer ):
+    if self._IsServerAlive():
+      return SendCommandRequest( arguments, completer )
+
+
+  def GetDefinedSubcommands( self ):
+    if self._IsServerAlive():
+      return BaseRequest.PostDataToHandler( BuildRequestData(),
+                                            'defined_subcommands' )
+    else:
+      return []
+
+
+  def GetCurrentCompletionRequest( self ):
+    return self._latest_completion_request
 
 
   def GetOmniCompleter( self ):
-    return self.omnicomp
+    return self._omnicomp
-
-
-  def GetFiletypeCompleter( self ):
-    filetypes = vimsupport.CurrentFiletypes()
-
-    completers = [ self.GetFiletypeCompleterForFiletype( filetype )
-                   for filetype in filetypes ]
-
-    if not completers:
-      return None
-
-    # Try to find a native completer first
-    for completer in completers:
-      if completer and completer is not self.omnicomp:
-        return completer
-
-    # Return the omni completer for the first filetype
-    return completers[0]
-
-
-  def GetFiletypeCompleterForFiletype( self, filetype ):
-    try:
-      return self.filetype_completers[ filetype ]
-    except KeyError:
-      pass
-
-    module_path = _PathToFiletypeCompleterPluginLoader( filetype )
-
-    completer = None
-    supported_filetypes = [ filetype ]
-    if os.path.exists( module_path ):
-      module = imp.load_source( filetype, module_path )
-      completer = module.GetCompleter()
-      if completer:
-        supported_filetypes.extend( completer.SupportedFiletypes() )
-    else:
-      completer = self.omnicomp
-
-    for supported_filetype in supported_filetypes:
-      self.filetype_completers[ supported_filetype ] = completer
-    return completer
-
-
-  def ShouldUseGeneralCompleter( self, start_column ):
-    return self.gencomp.ShouldUseNow( start_column )
-
-
-  def ShouldUseFiletypeCompleter( self, start_column ):
-    if self.FiletypeCompletionUsable():
-      return self.GetFiletypeCompleter().ShouldUseNow(
-        start_column )
-    return False
 
 
   def NativeFiletypeCompletionAvailable( self ):
-    completer = self.GetFiletypeCompleter()
+    return any( [ FiletypeCompleterExistsForFiletype( x ) for x in
-    return bool( completer ) and completer is not self.omnicomp
+                  vimsupport.CurrentFiletypes() ] )
-
-
-  def FiletypeCompletionAvailable( self ):
-    return bool( self.GetFiletypeCompleter() )
 
 
   def NativeFiletypeCompletionUsable( self ):
-    return ( _CurrentFiletypeCompletionEnabled() and
+    return ( self.CurrentFiletypeCompletionEnabled() and
              self.NativeFiletypeCompletionAvailable() )
 
 
-  def FiletypeCompletionUsable( self ):
-    return ( _CurrentFiletypeCompletionEnabled() and
-             self.FiletypeCompletionAvailable() )
-
-
   def OnFileReadyToParse( self ):
-    self.gencomp.OnFileReadyToParse()
+    self._omnicomp.OnFileReadyToParse( None )
 
-    if self.FiletypeCompletionUsable():
+    if not self._IsServerAlive():
-      self.GetFiletypeCompleter().OnFileReadyToParse()
+      self._NotifyUserIfServerCrashed()
+
+    extra_data = {}
+    if self._user_options[ 'collect_identifiers_from_tags_files' ]:
+      extra_data[ 'tag_files' ] = _GetTagFiles()
+
+    if self._user_options[ 'seed_identifiers_with_syntax' ]:
+      self._AddSyntaxDataIfNeeded( extra_data )
+
+    self._latest_file_parse_request = EventNotification( 'FileReadyToParse',
+                                                          extra_data )
+    self._latest_file_parse_request.Start()
 
 
   def OnBufferUnload( self, deleted_buffer_file ):
-    self.gencomp.OnBufferUnload( deleted_buffer_file )
+    if not self._IsServerAlive():
-
+      return
-    if self.FiletypeCompletionUsable():
+    SendEventNotificationAsync( 'BufferUnload',
-      self.GetFiletypeCompleter().OnBufferUnload( deleted_buffer_file )
+                                { 'unloaded_buffer': deleted_buffer_file } )
 
 
   def OnBufferVisit( self ):
-    self.gencomp.OnBufferVisit()
+    if not self._IsServerAlive():
-
+      return
-    if self.FiletypeCompletionUsable():
+    extra_data = {}
-      self.GetFiletypeCompleter().OnBufferVisit()
+    _AddUltiSnipsDataIfNeeded( extra_data )
+    SendEventNotificationAsync( 'BufferVisit', extra_data )
 
 
   def OnInsertLeave( self ):
-    self.gencomp.OnInsertLeave()
+    if not self._IsServerAlive():
-
+      return
-    if self.FiletypeCompletionUsable():
+    SendEventNotificationAsync( 'InsertLeave' )
-      self.GetFiletypeCompleter().OnInsertLeave()
 
 
   def OnVimLeave( self ):
-    self.gencomp.OnVimLeave()
+    if self._IsServerAlive():
+      self._server_popen.terminate()
+    os.remove( self._temp_options_filename )
 
-    if self.FiletypeCompletionUsable():
+    if not self._user_options[ 'server_keep_logfiles' ]:
-      self.GetFiletypeCompleter().OnVimLeave()
+      if self._server_stderr:
+        os.remove( self._server_stderr )
+      if self._server_stdout:
+        os.remove( self._server_stdout )
+
+
+  def OnCurrentIdentifierFinished( self ):
+    if not self._IsServerAlive():
+      return
+    SendEventNotificationAsync( 'CurrentIdentifierFinished' )
 
 
   def DiagnosticsForCurrentFileReady( self ):
-    if self.FiletypeCompletionUsable():
+    return bool( self._latest_file_parse_request and
-      return self.GetFiletypeCompleter().DiagnosticsForCurrentFileReady()
+                 self._latest_file_parse_request.Done() )
-    return False
 
 
-  def GetDiagnosticsForCurrentFile( self ):
+  def GetDiagnosticsFromStoredRequest( self ):
-    if self.FiletypeCompletionUsable():
+    if self.DiagnosticsForCurrentFileReady():
-      return self.GetFiletypeCompleter().GetDiagnosticsForCurrentFile()
+      to_return = self._latest_file_parse_request.Response()
+      # We set the diagnostics request to None because we want to prevent
+      # Syntastic from repeatedly refreshing the buffer with the same diags.
+      # Setting this to None makes DiagnosticsForCurrentFileReady return False
+      # until the next request is created.
+      self._latest_file_parse_request = None
+      return to_return
     return []
 
 
   def ShowDetailedDiagnostic( self ):
-    if self.FiletypeCompletionUsable():
+    if not self._IsServerAlive():
-      return self.GetFiletypeCompleter().ShowDetailedDiagnostic()
+      return
-
+    try:
-
+      debug_info = BaseRequest.PostDataToHandler( BuildRequestData(),
-  def GettingCompletions( self ):
+                                                  'detailed_diagnostic' )
-    if self.FiletypeCompletionUsable():
+      if 'message' in debug_info:
-      return self.GetFiletypeCompleter().GettingCompletions()
+        vimsupport.EchoText( debug_info[ 'message' ] )
-    return False
+    except ServerError as e:
-
+      vimsupport.PostVimMessage( str( e ) )
-
-  def OnCurrentIdentifierFinished( self ):
-    self.gencomp.OnCurrentIdentifierFinished()
-
-    if self.FiletypeCompletionUsable():
-      self.GetFiletypeCompleter().OnCurrentIdentifierFinished()
 
 
   def DebugInfo( self ):
-    completers = set( self.filetype_completers.values() )
+    if self._IsServerAlive():
-    completers.add( self.gencomp )
+      debug_info = BaseRequest.PostDataToHandler( BuildRequestData(),
-    output = []
+                                                  'debug_info' )
-    for completer in completers:
+    else:
-      if not completer:
+      debug_info = 'Server crashed, no debug info from server'
-        continue
+    debug_info += '\nServer running at: {0}'.format(
-      debug = completer.DebugInfo()
+        BaseRequest.server_location )
-      if debug:
+    debug_info += '\nServer process ID: {0}'.format( self._server_popen.pid )
-        output.append( debug )
+    if self._server_stderr or self._server_stdout:
+      debug_info += '\nServer logfiles:\n  {0}\n  {1}'.format(
+        self._server_stdout,
+        self._server_stderr )
 
-    has_clang_support = ycm_core.HasClangSupport()
+    return debug_info
-    output.append( 'Has Clang support compiled in: {0}'.format(
-      has_clang_support ) )
-
-    if has_clang_support:
-      output.append( ycm_core.ClangVersion() )
-
-    return '\n'.join( output )
 
 
-def _CurrentFiletypeCompletionEnabled():
+  def CurrentFiletypeCompletionEnabled( self ):
-  filetypes = vimsupport.CurrentFiletypes()
+    filetypes = vimsupport.CurrentFiletypes()
-  return not all([ x in FILETYPE_SPECIFIC_COMPLETION_TO_DISABLE
+    filetype_to_disable = self._user_options[
-                   for x in filetypes ])
+      'filetype_specific_completion_to_disable' ]
+    return not all([ x in filetype_to_disable for x in filetypes ])
 
 
-def _PathToCompletersFolder():
+  def _AddSyntaxDataIfNeeded( self, extra_data ):
+    filetype = vimsupport.CurrentFiletypes()[ 0 ]
+    if filetype in self._filetypes_with_keywords_loaded:
+      return
+
+    self._filetypes_with_keywords_loaded.add( filetype )
+    extra_data[ 'syntax_keywords' ] = list(
+       syntax_parse.SyntaxKeywordsForCurrentBuffer() )
+
+
+def _GetTagFiles():
+  tag_files = vim.eval( 'tagfiles()' )
+  current_working_directory = os.getcwd()
+  return [ os.path.join( current_working_directory, x ) for x in tag_files ]
+
+
+def _PathToServerScript():
   dir_of_current_script = os.path.dirname( os.path.abspath( __file__ ) )
-  return os.path.join( dir_of_current_script, 'completers' )
+  return os.path.join( dir_of_current_script, 'server/ycmd.py' )
 
 
-def _PathToFiletypeCompleterPluginLoader( filetype ):
+def _AddUltiSnipsDataIfNeeded( extra_data ):
-  return os.path.join( _PathToCompletersFolder(), filetype, 'hook.py' )
+  if not USE_ULTISNIPS_DATA:
+    return
 
+  try:
+    rawsnips = UltiSnips_Manager._snips( '', 1 )
+  except:
+    return
 
+  # UltiSnips_Manager._snips() returns a class instance where:
+  # class.trigger - name of snippet trigger word ( e.g. defn or testcase )
+  # class.description - description of the snippet
+  extra_data[ 'ultisnips_snippets' ] = [ { 'trigger': x.trigger,
+                                           'description': x.description
+                                         } for x in rawsnips ]

run_tests.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+set -e
+
+function usage {
+  echo "Usage: $0 [--no-clang-completer]"
+  exit 0
+}
+
+flake8 --select=F,C9 --max-complexity=10 python
+
+use_clang_completer=true
+for flag in $@; do
+  case "$flag" in
+    --no-clang-completer)
+      use_clang_completer=false
+      ;;
+    *)
+      usage
+      ;;
+  esac
+done
+
+if [ -n "$USE_CLANG_COMPLETER" ]; then
+  use_clang_completer=$USE_CLANG_COMPLETER
+fi
+
+if $use_clang_completer; then
+  extra_cmake_args="-DUSE_CLANG_COMPLETER=ON -DUSE_DEV_FLAGS=ON"
+else
+  extra_cmake_args="-DUSE_DEV_FLAGS=ON"
+fi
+
+EXTRA_CMAKE_ARGS=$extra_cmake_args YCM_TESTRUN=1 ./install.sh --omnisharp-completer
+
+for directory in third_party/*; do
+  if [ -d "${directory}" ]; then
+    export PYTHONPATH=$PWD/${directory}:$PYTHONPATH
+  fi
+done
+
+if $use_clang_completer; then
+  nosetests -v python
+else
+  nosetests -v --exclude=".*Clang.*" python
+fi

style_format.sh

@@ -19,6 +19,7 @@ astyle \
 --recursive \
 --exclude=gmock \
 --exclude=testdata \
+--exclude=ycm_client_support.cpp \
 --exclude=ycm_core.cpp \
 --exclude=CustomAssert.h \
 --exclude=CustomAssert.cpp \

third_party/argparse

@@ -0,0 +1 @@
+Subproject commit 46af816db4812eab5f4639717bf1ad2eb17cc1ff

third_party/bottle

@@ -0,0 +1 @@
+Subproject commit 154369b2b9f7393ca9d3d73de7e046e2342cf00a

third_party/frozendict

@@ -0,0 +1 @@
+Subproject commit 2ea45f3f429c5283f9f24738812f0ee152a8f4c1

third_party/jedi

@@ -0,0 +1 @@
+Subproject commit 099fe4eeb3544005a8e2ffdaae43fd8a12c82f16

third_party/pythonfutures/CHANGES

@@ -0,0 +1,44 @@
+2.1.4
+=====
+
+- Ported the library again from Python 3.2.5 to get the latest bug fixes 
+
+
+2.1.3
+=====
+
+- Fixed race condition in wait(return_when=ALL_COMPLETED)
+  (http://bugs.python.org/issue14406) -- thanks Ralf Schmitt
+- Added missing setUp() methods to several test classes
+
+
+2.1.2
+=====
+
+- Fixed installation problem on Python 3.1
+
+
+2.1.1
+=====
+
+- Fixed missing 'concurrent' package declaration in setup.py
+
+
+2.1
+===
+
+- Moved the code from the 'futures' package to 'concurrent.futures' to provide
+  a drop in backport that matches the code in Python 3.2 standard library
+- Deprecated the old 'futures' package
+
+
+2.0
+===
+
+- Changed implementation to match PEP 3148
+
+
+1.0
+===
+
+Initial release.

third_party/pythonfutures/LICENSE

@@ -0,0 +1,21 @@
+Copyright 2009 Brian Quinlan. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+   1. Redistributions of source code must retain the above copyright notice,
+      this list of conditions and the following disclaimer.
+   2. Redistributions in binary form must reproduce the above copyright notice,
+      this list of conditions and the following disclaimer in the documentation
+      and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY BRIAN QUINLAN "AS IS" AND ANY EXPRESS OR IMPLIED
+WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
+HALL THE FREEBSD PROJECT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

third_party/pythonfutures/concurrent/__init__.py

@@ -0,0 +1,3 @@
+from pkgutil import extend_path
+
+__path__ = extend_path(__path__, __name__)