diff --git a/.gitmodules b/.gitmodules index 7d13a22b..7dfea5ba 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,6 +1,24 @@ -[submodule "python/ycm/completers/python/jedi"] - path = python/ycm/completers/python/jedi - url = https://github.com/davidhalter/jedi.git +[submodule "third_party/jedi"] + path = third_party/jedi + url = https://github.com/davidhalter/jedi [submodule "python/ycm/completers/cs/OmniSharpServer"] path = python/ycm/completers/cs/OmniSharpServer url = https://github.com/nosami/OmniSharpServer.git +[submodule "third_party/requests-futures"] + path = third_party/requests-futures + url = https://github.com/ross/requests-futures +[submodule "third_party/requests"] + path = third_party/requests + url = https://github.com/kennethreitz/requests +[submodule "third_party/waitress"] + path = third_party/waitress + url = https://github.com/Pylons/waitress +[submodule "third_party/bottle"] + path = third_party/bottle + url = https://github.com/defnull/bottle +[submodule "third_party/frozendict"] + path = third_party/frozendict + url = https://github.com/slezica/python-frozendict +[submodule "third_party/argparse"] + path = third_party/argparse + url = https://github.com/bewest/argparse diff --git a/.travis.yml b/.travis.yml index fcec140b..7b1fa8d1 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,10 +4,11 @@ python: - "2.7" install: - pip install -r python/test_requirements.txt --use-mirrors + - sudo apt-get install mono-devel compiler: - gcc - clang -script: flake8 --exclude=jedi --select=F,C9 --max-complexity=10 python && ./install.sh && nosetests python +script: ./run_tests.sh env: - - YCM_TESTRUN=1 EXTRA_CMAKE_ARGS="" - - YCM_TESTRUN=1 EXTRA_CMAKE_ARGS="-DUSE_CLANG_COMPLETER=ON" + - USE_CLANG_COMPLETER="true" + - USE_CLANG_COMPLETER="false" diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 98e67062..c6bc2fb5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -44,14 +44,22 @@ Here are the things you should do when creating an issue: 1. **Write a step-by-step procedure that when performed repeatedly reproduces your issue.** If we can't reproduce the issue, then we can't fix it. It's that simple. -2. **Create a test case for your issue**. This is critical. Don't talk about how +2. Put the following options in your vimrc: + ``` + let g:ycm_server_use_vim_stdout = 1 + let g:ycm_server_log_level = 'debug' + ``` + Then start gvim/macvim (not console vim) from the console. As you use Vim, + you'll see the `ycmd` debug output stream in the console. Attach that to you + issue. +3. **Create a test case for your issue**. This is critical. Don't talk about how "when I have X in my file" or similar, _create a file with X in it_ and put the contents inside code blocks in your issue description. Try to make this test file _as small as possible_. Don't just paste a huge, 500 line source file you were editing and present that as a test. _Minimize_ the file so that the problem is reproduced with the smallest possible amount of test data. -3. **Include your OS and OS version.** -4. **Include the output of `vim --version`.** +4. **Include your OS and OS version.** +5. **Include the output of `vim --version`.** Creating good pull requests diff --git a/README.md b/README.md index 6c26afc8..8451d0ea 100644 --- a/README.md +++ b/README.md @@ -89,8 +89,9 @@ local binary folder (for example `/usr/local/bin/mvim`) and then symlink it: Install YouCompleteMe with [Vundle][]. **Remember:** YCM is a plugin with a compiled component. If you **update** YCM -using Vundle and the ycm_core library API has changed (happens rarely), YCM will -notify you to recompile it. You should then rerun the install process. +using Vundle and the ycm_support_libs library APIs have changed (happens +rarely), YCM will notify you to recompile it. You should then rerun the install +process. It's recommended that you have the latest Xcode installed along with the latest Command Line Tools (that you install from within Xcode). @@ -136,8 +137,9 @@ from source][vim-build] (don't worry, it's easy). Install YouCompleteMe with [Vundle][]. **Remember:** YCM is a plugin with a compiled component. If you **update** YCM -using Vundle and the ycm_core library API has changed (happens rarely), YCM will -notify you to recompile it. You should then rerun the install process. +using Vundle and the ycm_support_libs library APIs have changed (happens +rarely), YCM will notify you to recompile it. You should then rerun the install +process. Install development tools and CMake: `sudo apt-get install build-essential cmake` @@ -184,8 +186,9 @@ that platform). See the _FAQ_ if you have any issues. **Remember:** YCM is a plugin with a compiled component. If you **update** YCM -using Vundle and the ycm_core library API has changed (happens rarely), YCM will -notify you to recompile it. You should then rerun the install process. +using Vundle and the ycm_support_libs library APIs have changed (happens +rarely), YCM will notify you to recompile it. You should then rerun the install +process. **Please follow the instructions carefully. Read EVERY WORD.** @@ -221,8 +224,8 @@ notify you to recompile it. You should then rerun the install process. binaries from llvm.org][clang-download] if at all possible. Make sure you download the correct archive file for your OS. -4. **Compile the `ycm_core` plugin plugin** (ha!) that YCM needs. This is the - C++ engine that YCM uses to get fast completions. +4. **Compile the `ycm_support_libs` libraries** that YCM needs. These libs + are the C++ engines that YCM uses to get fast completions. You will need to have `cmake` installed in order to generate the required makefiles. Linux users can install cmake with their package manager (`sudo @@ -261,7 +264,7 @@ notify you to recompile it. You should then rerun the install process. Now that makefiles have been generated, simply run: - make ycm_core + make ycm_support_libs For those who want to use the system version of libclang, you would pass `-DUSE_SYSTEM_LIBCLANG=ON` to cmake _instead of_ the @@ -323,6 +326,13 @@ YCM automatically detects which completion engine would be the best in any situation. On occasion, it queries several of them at once, merges the outputs and presents the results to you. +### Client-server architecture + +YCM has a client-server architecture; the Vim part of YCM is only a thin client +that talks to the `ycmd` HTTP+JSON server that has the vast majority of YCM +logic and functionality. The server is started and stopped automatically as you +start and stop Vim. + ### Completion string ranking The subsequence filter removes any completions that do not match the input, but @@ -498,6 +508,11 @@ yours truly. Commands -------- +### The `:YcmRestartServer` command + +If the `ycmd` completion server suddenly stops for some reason, you can restart +it with this command. + ### The `:YcmForceCompileAndDiagnostics` command Calling this command will force YCM to immediately recompile your file @@ -832,6 +847,64 @@ Default: `0` let g:ycm_seed_identifiers_with_syntax = 0 +### The `g:ycm_server_use_vim_stdout` option + +By default, the `ycmd` completion server writes logs to logfiles. When this +option is set to `1`, the server writes logs to Vim's stdout (so you'll see them +in the console). + +Default: `0` + + let g:ycm_server_use_vim_stdout = 0 + +### The `g:ycm_server_keep_logfiles` option + +When this option is set to `1`, the `ycmd` completion server will keep the +logfiles around after shutting down (they are deleted on shutdown by default). + +To see where the logfiles are, call `:YcmDebugInfo`. + +Default: `0` + + let g:ycm_server_keep_logfiles = 0 + +### The `g:ycm_server_log_level` option + +The logging level that the `ycmd` completion server uses. Valid values are the +following, from most verbose to least verbose: +- `debug` +- `info` +- `warning` +- `error` +- `critical` + +Note that `debug` is _very_ verbose. + +Default: `info` + + let g:ycm_server_log_level = 'info' + +### The `g:ycm_server_idle_suicide_seconds` option + +This option sets the number of seconds of `ycmd` server idleness (no requests +received) after which the server stops itself. NOTE: the YCM Vim client sends a +shutdown request to the server when Vim is shutting down. + +If your Vim crashes for instance, `ycmd` never gets the shutdown command and +becomes a zombie process. This option prevents such zombies from sticking around +forever. + +The default option is `43200` seconds which is 12 hours. The reason for the +interval being this long is to prevent the server from shutting down if you +leave your computer (and Vim) turned on during the night. + +The server "heartbeat" that checks whether this interval has passed occurs every +10 minutes. + +Default: `43200` + + let g:ycm_server_idle_suicide_seconds = 43200 + ### The `g:ycm_csharp_server_port` option The port number (on `localhost`) on which the OmniSharp server should be diff --git a/autoload/youcompleteme.vim b/autoload/youcompleteme.vim index 51f6e1d1..4aa6feab 100644 --- a/autoload/youcompleteme.vim +++ b/autoload/youcompleteme.vim @@ -22,8 +22,6 @@ set cpo&vim " This needs to be called outside of a function let s:script_folder_path = escape( expand( ':p:h' ), '\' ) let s:searched_and_results_found = 0 -let s:should_use_filetype_completion = 0 -let s:completion_start_column = 0 let s:omnifunc_mode = 0 let s:old_cursor_position = [] @@ -31,28 +29,54 @@ let s:cursor_moved = 0 let s:moved_vertically_in_insert_mode = 0 let s:previous_num_chars_on_current_line = -1 +let s:forced_syntastic_checker_for = { + \ 'cpp': 1, + \ 'c': 1, + \ 'objc': 1, + \ 'objcpp': 1, + \ } + + function! youcompleteme#Enable() " When vim is in diff mode, don't run if &diff return endif + call s:SetUpBackwardsCompatibility() + py import sys py import vim exe 'python sys.path.insert( 0, "' . s:script_folder_path . '/../python" )' - py from ycm import extra_conf_store - py extra_conf_store.CallExtraConfYcmCorePreloadIfExists() + py from ycm import utils + py utils.AddThirdPartyFoldersToSysPath() py from ycm import base + py base.LoadJsonDefaultsIntoVim() + py from ycm import vimsupport + py from ycm import user_options_store + py user_options_store.SetAll( base.BuildServerConf() ) if !pyeval( 'base.CompatibleWithYcmCore()') echohl WarningMsg | - \ echomsg "YouCompleteMe unavailable: ycm_core too old, PLEASE RECOMPILE ycm_core" | + \ echomsg "YouCompleteMe unavailable: YCM support libs too old, PLEASE RECOMPILE" | \ echohl None return endif py from ycm.youcompleteme import YouCompleteMe - py ycm_state = YouCompleteMe() + py ycm_state = YouCompleteMe( user_options_store.GetAll() ) + + call s:SetUpCpoptions() + call s:SetUpCompleteopt() + call s:SetUpKeyMappings() + + if g:ycm_register_as_syntastic_checker + call s:ForceSyntasticCFamilyChecker() + endif + + if g:ycm_allow_changing_updatetime + set ut=2000 + endif augroup youcompleteme autocmd! @@ -63,7 +87,10 @@ function! youcompleteme#Enable() " is read. This is because youcompleteme#Enable() is called on VimEnter and " that happens *after" BufRead/BufEnter has already triggered for the " initial file. - autocmd BufRead,BufEnter * call s:OnBufferVisit() + " We also need to trigger buf init code on the FileType event because when + " the user does :enew and then :set ft=something, we need to run buf init + " code again. + autocmd BufRead,BufEnter,FileType * call s:OnBufferVisit() autocmd BufUnload * call s:OnBufferUnload( expand( ':p' ) ) autocmd CursorHold,CursorHoldI * call s:OnCursorHold() autocmd InsertLeave * call s:OnInsertLeave() @@ -71,19 +98,6 @@ function! youcompleteme#Enable() autocmd VimLeave * call s:OnVimLeave() augroup END - call s:SetUpCpoptions() - call s:SetUpCompleteopt() - call s:SetUpKeyMappings() - call s:SetUpBackwardsCompatibility() - - if g:ycm_register_as_syntastic_checker - call s:ForceSyntasticCFamilyChecker() - endif - - if g:ycm_allow_changing_updatetime - set ut=2000 - endif - " Calling this once solves the problem of BufRead/BufEnter not triggering for " the first loaded file. This should be the last command executed in this " function! @@ -151,6 +165,11 @@ function! s:SetUpBackwardsCompatibility() let g:ycm_complete_in_strings = 1 let g:ycm_complete_in_comments = 1 endif + + " ycm_filetypes_to_completely_ignore is the old name for fileype_blacklist + if has_key( g:, 'ycm_filetypes_to_completely_ignore' ) + let g:filetype_blacklist = g:ycm_filetypes_to_completely_ignore + endif endfunction @@ -163,6 +182,12 @@ function! s:ForceSyntasticCFamilyChecker() endfunction +function! s:ForcedAsSyntasticCheckerForCurrentFiletype() + return g:ycm_register_as_syntastic_checker && + \ get( s:forced_syntastic_checker_for, &filetype, 0 ) +endfunction + + function! s:AllowedToCompleteInCurrentFile() if empty( &filetype ) || getbufvar(winbufnr(winnr()), "&buftype") ==# 'nofile' return 0 @@ -221,7 +246,6 @@ endfunction function! s:OnVimLeave() py ycm_state.OnVimLeave() - py extra_conf_store.CallExtraConfVimCloseIfExists() endfunction @@ -257,9 +281,6 @@ function! s:OnCursorHold() endif call s:SetUpCompleteopt() - " Order is important here; we need to extract any done diagnostics before - " reparsing the file again - call s:UpdateDiagnosticNotifications() call s:OnFileReadyToParse() endfunction @@ -269,6 +290,12 @@ function! s:OnFileReadyToParse() " happen for special buffers. call s:SetUpYcmChangedTick() + " Order is important here; we need to extract any done diagnostics before + " reparsing the file again. If we sent the new parse request first, then + " the response would always be pending when we called + " UpdateDiagnosticNotifications. + call s:UpdateDiagnosticNotifications() + let buffer_changed = b:changedtick != b:ycm_changedtick.file_ready_to_parse if buffer_changed py ycm_state.OnFileReadyToParse() @@ -326,7 +353,6 @@ function! s:OnCursorMovedNormalMode() return endif - call s:UpdateDiagnosticNotifications() call s:OnFileReadyToParse() endfunction @@ -337,7 +363,6 @@ function! s:OnInsertLeave() endif let s:omnifunc_mode = 0 - call s:UpdateDiagnosticNotifications() call s:OnFileReadyToParse() py ycm_state.OnInsertLeave() if g:ycm_autoclose_preview_window_after_completion || @@ -407,10 +432,16 @@ endfunction function! s:UpdateDiagnosticNotifications() - if get( g:, 'loaded_syntastic_plugin', 0 ) && - \ pyeval( 'ycm_state.NativeFiletypeCompletionUsable()' ) && - \ pyeval( 'ycm_state.DiagnosticsForCurrentFileReady()' ) && - \ g:ycm_register_as_syntastic_checker + let should_display_diagnostics = + \ get( g:, 'loaded_syntastic_plugin', 0 ) && + \ s:ForcedAsSyntasticCheckerForCurrentFiletype() && + \ pyeval( 'ycm_state.NativeFiletypeCompletionUsable()' ) + + if !should_display_diagnostics + return + endif + + if pyeval( 'ycm_state.DiagnosticsForCurrentFileReady()' ) SyntasticCheck endif endfunction @@ -494,29 +525,24 @@ function! s:InvokeCompletion() endfunction -function! s:CompletionsForQuery( query, use_filetype_completer, - \ completion_start_column ) - if a:use_filetype_completer - py completer = ycm_state.GetFiletypeCompleter() - else - py completer = ycm_state.GetGeneralCompleter() - endif - - py completer.CandidatesForQueryAsync( vim.eval( 'a:query' ), - \ int( vim.eval( 'a:completion_start_column' ) ) ) - - let l:results_ready = 0 - while !l:results_ready - let l:results_ready = pyeval( 'completer.AsyncCandidateRequestReady()' ) - if complete_check() - let s:searched_and_results_found = 0 +python << EOF +def GetCompletions( query ): + request = ycm_state.GetCurrentCompletionRequest() + request.Start( query ) + while not request.Done(): + if bool( int( vim.eval( 'complete_check()' ) ) ): return { 'words' : [], 'refresh' : 'always'} - endif - endwhile - let l:results = pyeval( 'base.AdjustCandidateInsertionText( completer.CandidatesFromStoredRequest() )' ) - let s:searched_and_results_found = len( l:results ) != 0 - return { 'words' : l:results, 'refresh' : 'always' } + results = base.AdjustCandidateInsertionText( request.Response() ) + return { 'words' : results, 'refresh' : 'always' } +EOF + + +function! s:CompletionsForQuery( query ) + py results = GetCompletions( vim.eval( 'a:query' ) ) + let results = pyeval( 'results' ) + let s:searched_and_results_found = len( results.words ) != 0 + return results endfunction @@ -540,24 +566,13 @@ function! youcompleteme#Complete( findstart, base ) return -2 endif - - " TODO: make this a function-local variable instead of a script-local one - let s:completion_start_column = pyeval( 'base.CompletionStartColumn()' ) - let s:should_use_filetype_completion = - \ pyeval( 'ycm_state.ShouldUseFiletypeCompleter(' . - \ s:completion_start_column . ')' ) - - if !s:should_use_filetype_completion && - \ !pyeval( 'ycm_state.ShouldUseGeneralCompleter(' . - \ s:completion_start_column . ')' ) - " for vim, -2 means not found but don't trigger an error message - " see :h complete-functions + py request = ycm_state.CreateCompletionRequest() + if !pyeval( 'bool(request)' ) return -2 endif - return s:completion_start_column + return pyeval( 'request.CompletionStartColumn()' ) else - return s:CompletionsForQuery( a:base, s:should_use_filetype_completion, - \ s:completion_start_column ) + return s:CompletionsForQuery( a:base ) endif endfunction @@ -565,14 +580,26 @@ endfunction function! youcompleteme#OmniComplete( findstart, base ) if a:findstart let s:omnifunc_mode = 1 - let s:completion_start_column = pyeval( 'base.CompletionStartColumn()' ) - return s:completion_start_column + py request = ycm_state.CreateCompletionRequest( force_semantic = True ) + return pyeval( 'request.CompletionStartColumn()' ) else - return s:CompletionsForQuery( a:base, 1, s:completion_start_column ) + return s:CompletionsForQuery( a:base ) endif endfunction +function! youcompleteme#ServerPid() + return pyeval( 'ycm_state.ServerPid()' ) +endfunction + + +function! s:RestartServer() + py ycm_state.RestartServer() +endfunction + +command! YcmRestartServer call s:RestartServer() + + function! s:ShowDetailedDiagnostic() py ycm_state.ShowDetailedDiagnostic() endfunction @@ -584,7 +611,7 @@ command! YcmShowDetailedDiagnostic call s:ShowDetailedDiagnostic() " required (currently that's on buffer save) OR when the SyntasticCheck command " is invoked function! youcompleteme#CurrentFileDiagnostics() - return pyeval( 'ycm_state.GetDiagnosticsForCurrentFile()' ) + return pyeval( 'ycm_state.GetDiagnosticsFromStoredRequest()' ) endfunction @@ -598,37 +625,27 @@ endfunction command! YcmDebugInfo call s:DebugInfo() + function! s:CompleterCommand(...) " CompleterCommand will call the OnUserCommand function of a completer. " If the first arguments is of the form "ft=..." it can be used to specify the " completer to use (for example "ft=cpp"). Else the native filetype completer " of the current buffer is used. If no native filetype completer is found and - " no completer was specified this throws an error. You can use "ft=ycm:omni" - " to select the omni completer or "ft=ycm:ident" to select the identifier - " completer. The remaining arguments will passed to the completer. + " no completer was specified this throws an error. You can use + " "ft=ycm:ident" to select the identifier completer. + " The remaining arguments will be passed to the completer. let arguments = copy(a:000) + let completer = '' if a:0 > 0 && strpart(a:1, 0, 3) == 'ft=' - if a:1 == 'ft=ycm:omni' - py completer = ycm_state.GetOmniCompleter() - elseif a:1 == 'ft=ycm:ident' - py completer = ycm_state.GetGeneralCompleter() - else - py completer = ycm_state.GetFiletypeCompleterForFiletype( - \ vim.eval('a:1').lstrip('ft=') ) + if a:1 == 'ft=ycm:ident' + let completer = 'identifier' endif let arguments = arguments[1:] - elseif pyeval( 'ycm_state.NativeFiletypeCompletionAvailable()' ) - py completer = ycm_state.GetFiletypeCompleter() - else - echohl WarningMsg | - \ echomsg "No native completer found for current buffer." | - \ echomsg "Use ft=... as the first argument to specify a completer." | - \ echohl None - return endif - py completer.OnUserCommand( vim.eval( 'l:arguments' ) ) + py ycm_state.SendCommandRequest( vim.eval( 'l:arguments' ), + \ vim.eval( 'l:completer' ) ) endfunction @@ -645,9 +662,8 @@ endfunction command! -nargs=* -complete=custom,youcompleteme#SubCommandsComplete \ YcmCompleter call s:CompleterCommand() - function! youcompleteme#SubCommandsComplete( arglead, cmdline, cursorpos ) - return join( pyeval( 'ycm_state.GetFiletypeCompleter().DefinedSubcommands()' ), + return join( pyeval( 'ycm_state.GetDefinedSubcommands()' ), \ "\n") endfunction @@ -668,14 +684,6 @@ function! s:ForceCompile() break endif - let getting_completions = pyeval( - \ 'ycm_state.GettingCompletions()' ) - - if !getting_completions - echom "Unable to retrieve diagnostics, see output of `:mes` for possible details." - return 0 - endif - sleep 100m endwhile return 1 @@ -701,7 +709,7 @@ function! s:ShowDiagnostics() return endif - let diags = pyeval( 'ycm_state.GetDiagnosticsForCurrentFile()' ) + let diags = pyeval( 'ycm_state.GetDiagnosticsFromStoredRequest()' ) if !empty( diags ) call setloclist( 0, diags ) lopen diff --git a/cpp/BoostParts/CMakeLists.txt b/cpp/BoostParts/CMakeLists.txt index 0b1da46e..b2a648df 100644 --- a/cpp/BoostParts/CMakeLists.txt +++ b/cpp/BoostParts/CMakeLists.txt @@ -26,8 +26,8 @@ cmake_minimum_required( VERSION 2.8 ) project( BoostParts ) -set( Python_ADDITIONAL_VERSIONS 2.7 2.6 2.5 ) -find_package( PythonLibs 2.5 REQUIRED ) +set( Python_ADDITIONAL_VERSIONS 2.7 2.6 ) +find_package( PythonLibs 2.6 REQUIRED ) if ( NOT PYTHONLIBS_VERSION_STRING VERSION_LESS "3.0.0" ) message( FATAL_ERROR diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index 62873efb..5dbc4a2d 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -138,19 +138,18 @@ endif() # the compiler to output a warning during linking: # clang: warning: argument unused during compilation: '-std=c++0x' # This is caused by cmake passing this flag to the linking stage which it -# shouldn't do. It's ignored so it does no harm, but the warning is annoying and -# there's no way around the problem (the flag is correctly used during the -# compilation stage). We could use add_definitions(-std=c++0x), but this will -# break the llvm build since the flag will then be used when compiling C code -# too. Sadly there's no way around the warning. +# shouldn't do. It's ignored so it does no harm, but the warning is annoying. +# +# Putting the flag in add_definitions() works around the issue, even though it +# shouldn't in theory go there. if ( CPP11_AVAILABLE ) message( "Your C++ compiler supports C++11, compiling in that mode." ) # Cygwin needs its hand held a bit; see issue #473 if ( CYGWIN AND CMAKE_COMPILER_IS_GNUCXX ) - set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++0x" ) + add_definitions( -std=gnu++0x ) else() - set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x" ) + add_definitions( -std=c++0x ) endif() else() message( diff --git a/cpp/ycm/CMakeLists.txt b/cpp/ycm/CMakeLists.txt index 3c0654d8..fa7d30b1 100644 --- a/cpp/ycm/CMakeLists.txt +++ b/cpp/ycm/CMakeLists.txt @@ -17,10 +17,12 @@ cmake_minimum_required( VERSION 2.8 ) -project( ycm_core ) +project( ycm_support_libs ) +set( CLIENT_LIB "ycm_client_support" ) +set( SERVER_LIB "ycm_core" ) -set( Python_ADDITIONAL_VERSIONS 2.7 2.6 2.5 ) -find_package( PythonLibs 2.5 REQUIRED ) +set( Python_ADDITIONAL_VERSIONS 2.7 2.6 ) +find_package( PythonLibs 2.6 REQUIRED ) if ( NOT PYTHONLIBS_VERSION_STRING VERSION_LESS "3.0.0" ) message( FATAL_ERROR @@ -138,15 +140,15 @@ include_directories( ${CLANG_INCLUDES_DIR} ) -file( GLOB_RECURSE SOURCES *.h *.cpp ) +file( GLOB_RECURSE SERVER_SOURCES *.h *.cpp ) # The test sources are a part of a different target, so we remove them # The CMakeFiles cpp file is picked up when the user creates an in-source build, -# and we don't want that. -file( GLOB_RECURSE to_remove tests/*.h tests/*.cpp CMakeFiles/*.cpp ) +# and we don't want that. We also remove client-specific code +file( GLOB_RECURSE to_remove tests/*.h tests/*.cpp CMakeFiles/*.cpp *client* ) if( to_remove ) - list( REMOVE_ITEM SOURCES ${to_remove} ) + list( REMOVE_ITEM SERVER_SOURCES ${to_remove} ) endif() if ( USE_CLANG_COMPLETER ) @@ -158,7 +160,7 @@ else() file( GLOB_RECURSE to_remove_clang ClangCompleter/*.h ClangCompleter/*.cpp ) if( to_remove_clang ) - list( REMOVE_ITEM SOURCES ${to_remove_clang} ) + list( REMOVE_ITEM SERVER_SOURCES ${to_remove_clang} ) endif() endif() @@ -217,11 +219,33 @@ endif() ############################################################################# -add_library( ${PROJECT_NAME} SHARED - ${SOURCES} +# We don't actually need all of the files this picks up, just the ones needed by +# PythonSupport.cpp. But this is easier to maintain and dead code elemination +# will remove unused code. +file( GLOB CLIENT_SOURCES *.h *.cpp ) +file( GLOB SERVER_SPECIFIC *ycm_core* ) + +if( SERVER_SPECIFIC ) + list( REMOVE_ITEM CLIENT_SOURCES ${SERVER_SPECIFIC} ) +endif() + +add_library( ${CLIENT_LIB} SHARED + ${CLIENT_SOURCES} ) -target_link_libraries( ${PROJECT_NAME} +target_link_libraries( ${CLIENT_LIB} + BoostParts + ${PYTHON_LIBRARIES} + ${EXTRA_LIBS} + ) + +############################################################################# + +add_library( ${SERVER_LIB} SHARED + ${SERVER_SOURCES} + ) + +target_link_libraries( ${SERVER_LIB} BoostParts ${PYTHON_LIBRARIES} ${LIBCLANG_TARGET} @@ -231,35 +255,43 @@ target_link_libraries( ${PROJECT_NAME} if( LIBCLANG_TARGET ) if( NOT WIN32 ) add_custom_command( - TARGET ${PROJECT_NAME} + TARGET ${SERVER_LIB} POST_BUILD - COMMAND ${CMAKE_COMMAND} -E copy "${LIBCLANG_TARGET}" "$" + COMMAND ${CMAKE_COMMAND} -E copy "${LIBCLANG_TARGET}" "$" ) else() add_custom_command( - TARGET ${PROJECT_NAME} + TARGET ${SERVER_LIB} POST_BUILD - COMMAND ${CMAKE_COMMAND} -E copy "${PATH_TO_LLVM_ROOT}/bin/libclang.dll" "$") + COMMAND ${CMAKE_COMMAND} -E copy "${PATH_TO_LLVM_ROOT}/bin/libclang.dll" "$") endif() endif() +############################################################################# + +# Convenience target that builds both support libs. +add_custom_target( ${PROJECT_NAME} + DEPENDS ${CLIENT_LIB} ${SERVER_LIB} ) + + ############################################################################# # Things are a bit different on Macs when using an external libclang.dylib; here # we want to make sure we use @loader_path/libclang.dylib instead of -# @rpath/libclang.dylib in the final ycm_core.so. If we use the @rpath version, -# then it may load the system libclang which the user explicitely does not want -# (otherwise the user would specify USE_SYSTEM_LIBCLANG). With @loader_path, we -# make sure that only the libclang.dylib present in the same directory as our -# ycm_core.so is used. +# @rpath/libclang.dylib in the final ycm_core.so. If we use the +# @rpath version, then it may load the system libclang which the user +# explicitely does not want (otherwise the user would specify +# USE_SYSTEM_LIBCLANG). With @loader_path, we make sure that only the +# libclang.dylib present in the same directory as our ycm_core.so +# is used. if ( EXTERNAL_LIBCLANG_PATH AND APPLE ) - add_custom_command( TARGET ${PROJECT_NAME} + add_custom_command( TARGET ${SERVER_LIB} POST_BUILD COMMAND install_name_tool "-change" "@rpath/libclang.dylib" "@loader_path/libclang.dylib" - "$" + "$" ) endif() @@ -268,19 +300,24 @@ endif() # We don't want the "lib" prefix, it can screw up python when it tries to search # for our module -set_target_properties( ${PROJECT_NAME} PROPERTIES PREFIX "") +set_target_properties( ${CLIENT_LIB} PROPERTIES PREFIX "") +set_target_properties( ${SERVER_LIB} PROPERTIES PREFIX "") if ( WIN32 OR CYGWIN ) # This is the extension for compiled Python modules on Windows - set_target_properties( ${PROJECT_NAME} PROPERTIES SUFFIX ".pyd") + set_target_properties( ${CLIENT_LIB} PROPERTIES SUFFIX ".pyd") + set_target_properties( ${SERVER_LIB} PROPERTIES SUFFIX ".pyd") else() # Even on macs, we want a .so extension instead of a .dylib which is what # cmake would give us by default. Python won't recognize a .dylib as a module, # but it will recognize a .so - set_target_properties( ${PROJECT_NAME} PROPERTIES SUFFIX ".so") + set_target_properties( ${CLIENT_LIB} PROPERTIES SUFFIX ".so") + set_target_properties( ${SERVER_LIB} PROPERTIES SUFFIX ".so") endif() -set_target_properties( ${PROJECT_NAME} PROPERTIES +set_target_properties( ${CLIENT_LIB} PROPERTIES + LIBRARY_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/../../python ) +set_target_properties( ${SERVER_LIB} PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/../../python ) ############################################################################# diff --git a/cpp/ycm/ClangCompleter/ClangCompleter.cpp b/cpp/ycm/ClangCompleter/ClangCompleter.cpp index f822dbbd..7c4a74ab 100644 --- a/cpp/ycm/ClangCompleter/ClangCompleter.cpp +++ b/cpp/ycm/ClangCompleter/ClangCompleter.cpp @@ -1,4 +1,4 @@ -// Copyright (C) 2011, 2012 Strahinja Val Markovic +// Copyright (C) 2011-2013 Strahinja Val Markovic // // This file is part of YouCompleteMe. // @@ -23,48 +23,22 @@ #include "standard.h" #include "CandidateRepository.h" #include "CompletionData.h" -#include "ConcurrentLatestValue.h" #include "Utils.h" #include "ClangUtils.h" +#include "ReleaseGil.h" #include -#include -#include -#include +#include -using boost::algorithm::any_of; -using boost::algorithm::is_upper; -using boost::bind; -using boost::cref; -using boost::function; -using boost::lock_guard; -using boost::make_shared; -using boost::mutex; -using boost::packaged_task; -using boost::ref; -using boost::shared_lock; -using boost::shared_mutex; using boost::shared_ptr; -using boost::thread; -using boost::thread_interrupted; -using boost::try_to_lock_t; -using boost::unique_future; -using boost::unique_lock; using boost::unordered_map; namespace YouCompleteMe { -extern const unsigned int MAX_ASYNC_THREADS; -extern const unsigned int MIN_ASYNC_THREADS; - ClangCompleter::ClangCompleter() : clang_index_( clang_createIndex( 0, 0 ) ), - translation_unit_store_( clang_index_ ), - candidate_repository_( CandidateRepository::Instance() ), - threading_enabled_( false ), - time_to_die_( false ), - clang_data_ready_( false ) { + translation_unit_store_( clang_index_ ) { // The libclang docs don't say what is the default value for crash recovery. // I'm pretty sure it's turned on by default, but I'm not going to take any // chances. @@ -73,47 +47,20 @@ ClangCompleter::ClangCompleter() ClangCompleter::~ClangCompleter() { - { - unique_lock< shared_mutex > lock( time_to_die_mutex_ ); - time_to_die_ = true; - } - - sorting_threads_.interrupt_all(); - sorting_threads_.join_all(); - - if ( clang_thread_ ) { - clang_thread_->interrupt(); - clang_thread_->join(); - } - // We need to destroy all TUs before calling clang_disposeIndex because // the translation units need to be destroyed before the index is destroyed. + // Technically, a thread could still be holding onto a shared_ptr object + // when we destroy the clang index, but since we're shutting down, we don't + // really care. + // In practice, this situation shouldn't happen because the server threads are + // Python deamon threads and will all be killed before the main thread exits. translation_unit_store_.RemoveAll(); clang_disposeIndex( clang_index_ ); } -// We need this mostly so that we can not use it in tests. Apparently the -// GoogleTest framework goes apeshit on us (on some platforms, in some -// occasions) if we enable threads by default. -void ClangCompleter::EnableThreading() { - threading_enabled_ = true; - InitThreads(); -} - - -std::vector< Diagnostic > ClangCompleter::DiagnosticsForFile( - const std::string &filename ) { - shared_ptr< TranslationUnit > unit = translation_unit_store_.Get( filename ); - - if ( !unit ) - return std::vector< Diagnostic >(); - - return unit->LatestDiagnostics(); -} - - bool ClangCompleter::UpdatingTranslationUnit( const std::string &filename ) { + ReleaseGil unlock; shared_ptr< TranslationUnit > unit = translation_unit_store_.Get( filename ); if ( !unit ) @@ -126,10 +73,11 @@ bool ClangCompleter::UpdatingTranslationUnit( const std::string &filename ) { } -void ClangCompleter::UpdateTranslationUnit( +std::vector< Diagnostic > ClangCompleter::UpdateTranslationUnit( const std::string &filename, const std::vector< UnsavedFile > &unsaved_files, const std::vector< std::string > &flags ) { + ReleaseGil unlock; bool translation_unit_created; shared_ptr< TranslationUnit > unit = translation_unit_store_.GetOrCreate( filename, @@ -138,13 +86,14 @@ void ClangCompleter::UpdateTranslationUnit( translation_unit_created ); if ( !unit ) - return; + return std::vector< Diagnostic >(); try { // There's no point in reparsing a TU that was just created, it was just // parsed in the TU constructor if ( !translation_unit_created ) - unit->Reparse( unsaved_files ); + return unit->Reparse( unsaved_files ); + return unit->LatestDiagnostics(); } catch ( ClangParseError & ) { @@ -153,30 +102,8 @@ void ClangCompleter::UpdateTranslationUnit( // TU map. translation_unit_store_.Remove( filename ); } -} - -Future< void > ClangCompleter::UpdateTranslationUnitAsync( - std::string filename, - std::vector< UnsavedFile > unsaved_files, - std::vector< std::string > flags ) { - function< void() > functor = - boost::bind( &ClangCompleter::UpdateTranslationUnit, - boost::ref( *this ), - boost::move( filename ), - boost::move( unsaved_files ), - boost::move( flags ) ); - - shared_ptr< ClangPackagedTask > clang_packaged_task = - make_shared< ClangPackagedTask >(); - - clang_packaged_task->parsing_task_ = packaged_task< void >( - boost::move( functor ) ); - unique_future< void > future = - clang_packaged_task->parsing_task_.get_future(); - clang_task_.Set( clang_packaged_task ); - - return Future< void >( boost::move( future ) ); + return std::vector< Diagnostic >(); } @@ -187,6 +114,7 @@ ClangCompleter::CandidatesForLocationInFile( int column, const std::vector< UnsavedFile > &unsaved_files, const std::vector< std::string > &flags ) { + ReleaseGil unlock; shared_ptr< TranslationUnit > unit = translation_unit_store_.GetOrCreate( filename, unsaved_files, flags ); @@ -199,58 +127,13 @@ ClangCompleter::CandidatesForLocationInFile( } -Future< AsyncCompletions > -ClangCompleter::CandidatesForQueryAndLocationInFileAsync( - const std::string &query, - const std::string &filename, - int line, - int column, - const std::vector< UnsavedFile > &unsaved_files, - const std::vector< std::string > &flags ) { - // TODO: throw exception when threading is not enabled and this is called - if ( !threading_enabled_ ) - return Future< AsyncCompletions >(); - - bool skip_clang_result_cache = ShouldSkipClangResultCache( query, - line, - column ); - - if ( skip_clang_result_cache ) { - // The clang thread is busy, return nothing - if ( UpdatingTranslationUnit( filename ) ) - return Future< AsyncCompletions >(); - - { - lock_guard< mutex > lock( clang_data_ready_mutex_ ); - clang_data_ready_ = false; - } - - // Needed to "reset" the sorting threads to the start of their loop. This - // way any threads blocking on a read in sorting_task_.Get() are reset to - // wait on the clang_data_ready_condition_variable_. - sorting_threads_.interrupt_all(); - } - - // the sorting task needs to be set before the clang task (if any) just in - // case the clang task finishes (and therefore notifies a sorting thread to - // consume a sorting task) before the sorting task is set - unique_future< AsyncCompletions > future; - CreateSortingTask( query, future ); - - if ( skip_clang_result_cache ) { - CreateClangTask( filename, line, column, unsaved_files, flags ); - } - - return Future< AsyncCompletions >( boost::move( future ) ); -} - - Location ClangCompleter::GetDeclarationLocation( const std::string &filename, int line, int column, const std::vector< UnsavedFile > &unsaved_files, const std::vector< std::string > &flags ) { + ReleaseGil unlock; shared_ptr< TranslationUnit > unit = translation_unit_store_.GetOrCreate( filename, unsaved_files, flags ); @@ -268,6 +151,7 @@ Location ClangCompleter::GetDefinitionLocation( int column, const std::vector< UnsavedFile > &unsaved_files, const std::vector< std::string > &flags ) { + ReleaseGil unlock; shared_ptr< TranslationUnit > unit = translation_unit_store_.GetOrCreate( filename, unsaved_files, flags ); @@ -279,216 +163,9 @@ Location ClangCompleter::GetDefinitionLocation( } -void ClangCompleter::DeleteCachesForFileAsync( const std::string &filename ) { - file_cache_delete_stack_.Push( filename ); -} - - -void ClangCompleter::DeleteCaches() { - std::vector< std::string > filenames; - - if ( !file_cache_delete_stack_.PopAllNoWait( filenames ) ) - return; - - foreach( const std::string & filename, filenames ) { - translation_unit_store_.Remove( filename ); - } -} - - -bool ClangCompleter::ShouldSkipClangResultCache( const std::string &query, - int line, - int column ) { - // We need query.empty() in addition to the second check because if we don't - // have it, then we have a problem in the following situation: - // The user has a variable 'foo' of type 'A' and a variable 'bar' of type 'B'. - // He then types in 'foo.' and we store the clang results and also return - // them. The user then deletes that code and types in 'bar.'. Without the - // query.empty() check we would return the results from the cache here (it's - // the same line and column!) which would be incorrect. - return query.empty() || latest_clang_results_ - .NewPositionDifferentFromStoredPosition( line, column ); -} - - -// Copy-ctor for unique_future is private in C++03 mode so we need to take it as -// an out param -void ClangCompleter::CreateSortingTask( - const std::string &query, - unique_future< AsyncCompletions > &future ) { - // Careful! The code in this function may burn your eyes. - - function< CompletionDatas( const CompletionDatas & ) > - sort_candidates_for_query_functor = - boost::bind( &ClangCompleter::SortCandidatesForQuery, - boost::ref( *this ), - query, - _1 ); - - function< CompletionDatas() > operate_on_completion_data_functor = - boost::bind( &ClangResultsCache::OperateOnCompletionDatas< CompletionDatas >, - boost::cref( latest_clang_results_ ), - boost::move( sort_candidates_for_query_functor ) ); - - shared_ptr< packaged_task< AsyncCompletions > > task = - boost::make_shared< packaged_task< AsyncCompletions > >( - boost::bind( ReturnValueAsShared< std::vector< CompletionData > >, - boost::move( operate_on_completion_data_functor ) ) ); - - future = task->get_future(); - sorting_task_.Set( task ); -} - - -void ClangCompleter::CreateClangTask( - std::string filename, - int line, - int column, - std::vector< UnsavedFile > unsaved_files, - std::vector< std::string > flags ) { - latest_clang_results_.ResetWithNewLineAndColumn( line, column ); - - function< CompletionDatas() > candidates_for_location_functor = - boost::bind( &ClangCompleter::CandidatesForLocationInFile, - boost::ref( *this ), - boost::move( filename ), - line, - column, - boost::move( unsaved_files ), - boost::move( flags ) ); - - shared_ptr< ClangPackagedTask > clang_packaged_task = - make_shared< ClangPackagedTask >(); - - clang_packaged_task->completions_task_ = - packaged_task< AsyncCompletions >( - boost::bind( ReturnValueAsShared< std::vector< CompletionData > >, - boost::move( candidates_for_location_functor ) ) ); - - clang_task_.Set( clang_packaged_task ); -} - - - -std::vector< CompletionData > ClangCompleter::SortCandidatesForQuery( - const std::string &query, - const std::vector< CompletionData > &completion_datas ) { - Bitset query_bitset = LetterBitsetFromString( query ); - bool query_has_uppercase_letters = any_of( query, is_upper() ); - - std::vector< const Candidate * > repository_candidates = - candidate_repository_.GetCandidatesForStrings( completion_datas ); - - std::vector< ResultAnd< CompletionData * > > data_and_results; - - for ( uint i = 0; i < repository_candidates.size(); ++i ) { - const Candidate *candidate = repository_candidates[ i ]; - - if ( !candidate->MatchesQueryBitset( query_bitset ) ) - continue; - - Result result = candidate->QueryMatchResult( query, - query_has_uppercase_letters ); - - if ( result.IsSubsequence() ) { - ResultAnd< CompletionData * > data_and_result( &completion_datas[ i ], - result ); - data_and_results.push_back( boost::move( data_and_result ) ); - } - } - - std::sort( data_and_results.begin(), data_and_results.end() ); - - std::vector< CompletionData > sorted_completion_datas; - sorted_completion_datas.reserve( data_and_results.size() ); - - foreach ( const ResultAnd< CompletionData * > &data_and_result, - data_and_results ) { - sorted_completion_datas.push_back( *data_and_result.extra_object_ ); - } - - return sorted_completion_datas; -} - - -void ClangCompleter::InitThreads() { - int threads_to_create = - std::max( MIN_ASYNC_THREADS, - std::min( MAX_ASYNC_THREADS, thread::hardware_concurrency() ) ); - - for ( int i = 0; i < threads_to_create; ++i ) { - sorting_threads_.create_thread( bind( &ClangCompleter::SortingThreadMain, - boost::ref( *this ) ) ); - } - - clang_thread_.reset( new thread( &ClangCompleter::ClangThreadMain, - boost::ref( *this ) ) ); -} - - -void ClangCompleter::ClangThreadMain() { - while ( true ) { - try { - shared_ptr< ClangPackagedTask > task = clang_task_.Get(); - - bool has_completions_task = task->completions_task_.valid(); - - if ( has_completions_task ) - task->completions_task_(); - else - task->parsing_task_(); - - if ( !has_completions_task ) { - DeleteCaches(); - continue; - } - - unique_future< AsyncCompletions > future = - task->completions_task_.get_future(); - latest_clang_results_.SetCompletionDatas( *future.get() ); - - { - lock_guard< mutex > lock( clang_data_ready_mutex_ ); - clang_data_ready_ = true; - } - - clang_data_ready_condition_variable_.notify_all(); - } catch ( thread_interrupted & ) { - shared_lock< shared_mutex > lock( time_to_die_mutex_ ); - - if ( time_to_die_ ) - return; - - // else do nothing and re-enter the loop - } - } -} - - -void ClangCompleter::SortingThreadMain() { - while ( true ) { - try { - { - unique_lock< mutex > lock( clang_data_ready_mutex_ ); - - while ( !clang_data_ready_ ) { - clang_data_ready_condition_variable_.wait( lock ); - } - } - - shared_ptr< packaged_task< AsyncCompletions > > task = - sorting_task_.Get(); - - ( *task )(); - } catch ( thread_interrupted & ) { - shared_lock< shared_mutex > lock( time_to_die_mutex_ ); - - if ( time_to_die_ ) - return; - - // else do nothing and re-enter the loop - } - } +void ClangCompleter::DeleteCachesForFile( const std::string &filename ) { + ReleaseGil unlock; + translation_unit_store_.Remove( filename ); } diff --git a/cpp/ycm/ClangCompleter/ClangCompleter.h b/cpp/ycm/ClangCompleter/ClangCompleter.h index 266e3122..7f89019e 100644 --- a/cpp/ycm/ClangCompleter/ClangCompleter.h +++ b/cpp/ycm/ClangCompleter/ClangCompleter.h @@ -18,18 +18,11 @@ #ifndef CLANGCOMPLETE_H_WLKDU0ZV #define CLANGCOMPLETE_H_WLKDU0ZV -#include "ConcurrentLatestValue.h" -#include "ConcurrentStack.h" -#include "Future.h" #include "UnsavedFile.h" #include "Diagnostic.h" -#include "ClangResultsCache.h" #include "TranslationUnitStore.h" #include -#include -#include -#include #include @@ -37,46 +30,26 @@ typedef struct CXTranslationUnitImpl *CXTranslationUnit; namespace YouCompleteMe { -class CandidateRepository; class TranslationUnit; struct CompletionData; struct Location; typedef std::vector< CompletionData > CompletionDatas; -typedef boost::shared_ptr< CompletionDatas > AsyncCompletions; -typedef boost::unordered_map < std::string, - boost::shared_ptr < - std::vector< std::string > > > FlagsForFile; - - -// TODO: document that all filename parameters must be absolute paths +// All filename parameters must be absolute paths. class ClangCompleter : boost::noncopyable { public: ClangCompleter(); ~ClangCompleter(); - void EnableThreading(); - - std::vector< Diagnostic > DiagnosticsForFile( const std::string &filename ); - bool UpdatingTranslationUnit( const std::string &filename ); - // Public because of unit tests (gtest is not very thread-friendly) - void UpdateTranslationUnit( + std::vector< Diagnostic > UpdateTranslationUnit( const std::string &filename, const std::vector< UnsavedFile > &unsaved_files, const std::vector< std::string > &flags ); - // NOTE: params are taken by value on purpose! With a C++11 compiler we can - // avoid internal copies if params are taken by value (move ctors FTW) - Future< void > UpdateTranslationUnitAsync( - std::string filename, - std::vector< UnsavedFile > unsaved_files, - std::vector< std::string > flags ); - - // Public because of unit tests (gtest is not very thread-friendly) std::vector< CompletionData > CandidatesForLocationInFile( const std::string &filename, int line, @@ -84,14 +57,6 @@ public: const std::vector< UnsavedFile > &unsaved_files, const std::vector< std::string > &flags ); - Future< AsyncCompletions > CandidatesForQueryAndLocationInFileAsync( - const std::string &query, - const std::string &filename, - int line, - int column, - const std::vector< UnsavedFile > &unsaved_files, - const std::vector< std::string > &flags ); - Location GetDeclarationLocation( const std::string &filename, int line, @@ -106,55 +71,10 @@ public: const std::vector< UnsavedFile > &unsaved_files, const std::vector< std::string > &flags ); - void DeleteCachesForFileAsync( const std::string &filename ); + void DeleteCachesForFile( const std::string &filename ); private: - void DeleteCaches(); - - // This is basically a union. Only one of the two tasks is set to something - // valid, the other task is invalid. Which one is valid depends on the caller. - struct ClangPackagedTask { - boost::packaged_task< AsyncCompletions > completions_task_; - boost::packaged_task< void > parsing_task_; - }; - - typedef ConcurrentLatestValue < - boost::shared_ptr < - boost::packaged_task< AsyncCompletions > > > LatestSortingTask; - - typedef ConcurrentLatestValue < - boost::shared_ptr< ClangPackagedTask > > LatestClangTask; - - typedef ConcurrentStack< std::string > FileCacheDeleteStack; - - bool ShouldSkipClangResultCache( const std::string &query, - int line, - int column ); - - void CreateSortingTask( const std::string &query, - boost::unique_future< AsyncCompletions > &future ); - - // NOTE: params are taken by value on purpose! With a C++11 compiler we can - // avoid internal copies if params are taken by value (move ctors FTW) - void CreateClangTask( - std::string filename, - int line, - int column, - std::vector< UnsavedFile > unsaved_files, - std::vector< std::string > flags ); - - std::vector< CompletionData > SortCandidatesForQuery( - const std::string &query, - const std::vector< CompletionData > &completion_datas ); - - void InitThreads(); - - void ClangThreadMain(); - - void SortingThreadMain(); - - ///////////////////////////// // PRIVATE MEMBER VARIABLES ///////////////////////////// @@ -162,35 +82,6 @@ private: CXIndex clang_index_; TranslationUnitStore translation_unit_store_; - - CandidateRepository &candidate_repository_; - - bool threading_enabled_; - - // TODO: use boost.atomic for time_to_die_ - bool time_to_die_; - boost::shared_mutex time_to_die_mutex_; - - // TODO: use boost.atomic for clang_data_ready_ - bool clang_data_ready_; - boost::mutex clang_data_ready_mutex_; - boost::condition_variable clang_data_ready_condition_variable_; - - ClangResultsCache latest_clang_results_; - - FileCacheDeleteStack file_cache_delete_stack_; - - // Unfortunately clang is not thread-safe so we need to be careful when we - // access it. Only one thread at a time is allowed to access any single - // translation unit. Currently we only use one thread to access clang and that - // is the thread represented by clang_thread_. - boost::scoped_ptr< boost::thread > clang_thread_; - - boost::thread_group sorting_threads_; - - mutable LatestClangTask clang_task_; - - mutable LatestSortingTask sorting_task_; }; } // namespace YouCompleteMe diff --git a/cpp/ycm/ClangCompleter/ClangResultsCache.h b/cpp/ycm/ClangCompleter/ClangResultsCache.h deleted file mode 100644 index cae4f5fc..00000000 --- a/cpp/ycm/ClangCompleter/ClangResultsCache.h +++ /dev/null @@ -1,84 +0,0 @@ -// Copyright (C) 2011, 2012 Strahinja Val Markovic -// -// This file is part of YouCompleteMe. -// -// YouCompleteMe is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// YouCompleteMe is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with YouCompleteMe. If not, see . - -#ifndef CLANGRESULTSCACHE_H_REUWM3RU -#define CLANGRESULTSCACHE_H_REUWM3RU - -#include "CompletionData.h" - -#include -#include -#include -#include -#include -#include - -namespace YouCompleteMe { - -struct CompletionData; - -class ClangResultsCache : boost::noncopyable { -public: - - ClangResultsCache() : line_( -1 ), column_( -1 ) {} - - bool NewPositionDifferentFromStoredPosition( int new_line, int new_colum ) - const; - - void ResetWithNewLineAndColumn( int new_line, int new_colum ); - - void SetCompletionDatas( - const std::vector< CompletionData > new_completions ) { - completion_datas_ = new_completions; - } - -#ifndef BOOST_NO_RVALUE_REFERENCES -# ifdef __clang__ -# pragma clang diagnostic push -# pragma clang diagnostic ignored "-Wc++98-compat" -# endif //#ifdef __clang__ - - void SetCompletionDatas( std::vector< CompletionData > &&new_completions ) { - completion_datas_ = new_completions; - } - -# ifdef __clang__ -# pragma clang diagnostic pop -# endif //#ifdef __clang__ -#endif //#ifndef BOOST_NO_RVALUE_REFERENCES - - template< typename T > - T OperateOnCompletionDatas( - boost::function< T( const std::vector< CompletionData >& ) > operation ) - const { - boost::shared_lock< boost::shared_mutex > reader_lock( access_mutex_ ); - return operation( completion_datas_ ); - } - -private: - - int line_; - int column_; - std::vector< CompletionData > completion_datas_; - - mutable boost::shared_mutex access_mutex_; -}; - -} // namespace YouCompleteMe - -#endif /* end of include guard: CLANGRESULTSCACHE_H_REUWM3RU */ - diff --git a/cpp/ycm/ClangCompleter/ClangUtils.cpp b/cpp/ycm/ClangCompleter/ClangUtils.cpp index 30ec2767..cf4445c6 100644 --- a/cpp/ycm/ClangCompleter/ClangUtils.cpp +++ b/cpp/ycm/ClangCompleter/ClangUtils.cpp @@ -112,11 +112,8 @@ std::vector< CXUnsavedFile > ToCXUnsavedFiles( std::vector< CXUnsavedFile > clang_unsaved_files( unsaved_files.size() ); for ( uint i = 0; i < unsaved_files.size(); ++i ) { - X_VERIFY( unsaved_files[ i ].filename_ ); - X_VERIFY( unsaved_files[ i ].contents_ ); - X_VERIFY( unsaved_files[ i ].length_ ); - clang_unsaved_files[ i ].Filename = unsaved_files[ i ].filename_; - clang_unsaved_files[ i ].Contents = unsaved_files[ i ].contents_; + clang_unsaved_files[ i ].Filename = unsaved_files[ i ].filename_.c_str(); + clang_unsaved_files[ i ].Contents = unsaved_files[ i ].contents_.c_str(); clang_unsaved_files[ i ].Length = unsaved_files[ i ].length_; } diff --git a/cpp/ycm/ClangCompleter/ClangUtils.h b/cpp/ycm/ClangCompleter/ClangUtils.h index b7dd7a43..088c43b2 100644 --- a/cpp/ycm/ClangCompleter/ClangUtils.h +++ b/cpp/ycm/ClangCompleter/ClangUtils.h @@ -37,6 +37,8 @@ std::string CXStringToString( CXString text ); std::vector< CompletionData > ToCompletionDataVector( CXCodeCompleteResults *results ); +// NOTE: CXUnsavedFiles store pointers to data in UnsavedFiles, so UnsavedFiles +// need to outlive CXUnsavedFiles! std::vector< CXUnsavedFile > ToCXUnsavedFiles( const std::vector< UnsavedFile > &unsaved_files ); diff --git a/cpp/ycm/ClangCompleter/CompilationDatabase.cpp b/cpp/ycm/ClangCompleter/CompilationDatabase.cpp index 35444878..249d9608 100644 --- a/cpp/ycm/ClangCompleter/CompilationDatabase.cpp +++ b/cpp/ycm/ClangCompleter/CompilationDatabase.cpp @@ -18,20 +18,19 @@ #include "CompilationDatabase.h" #include "ClangUtils.h" #include "standard.h" +#include "ReleaseGil.h" #include -#include #include #include +#include -using boost::bind; -using boost::make_shared; -using boost::packaged_task; +using boost::lock_guard; +using boost::unique_lock; +using boost::try_to_lock_t; using boost::remove_pointer; using boost::shared_ptr; -using boost::thread; -using boost::unique_future; -using boost::function; +using boost::mutex; namespace YouCompleteMe { @@ -39,22 +38,9 @@ typedef shared_ptr < remove_pointer< CXCompileCommands >::type > CompileCommandsWrap; -void QueryThreadMain( CompilationDatabase::InfoTaskStack &info_task_stack ) { - while ( true ) { - try { - ( *info_task_stack.Pop() )(); - } catch ( boost::thread_interrupted & ) { - return; - } - } - -} - - CompilationDatabase::CompilationDatabase( const std::string &path_to_directory ) - : threading_enabled_( false ), - is_loaded_( false ) { + : is_loaded_( false ) { CXCompilationDatabase_Error status; compilation_database_ = clang_CompilationDatabase_fromDirectory( path_to_directory.c_str(), @@ -68,27 +54,26 @@ CompilationDatabase::~CompilationDatabase() { } -// We need this mostly so that we can not use it in tests. Apparently the -// GoogleTest framework goes apeshit on us if we enable threads by default. -void CompilationDatabase::EnableThreading() { - threading_enabled_ = true; - InitThreads(); -} - - bool CompilationDatabase::DatabaseSuccessfullyLoaded() { return is_loaded_; } +bool CompilationDatabase::AlreadyGettingFlags() { + unique_lock< mutex > lock( compilation_database_mutex_, try_to_lock_t() ); + return !lock.owns_lock(); +} + + CompilationInfoForFile CompilationDatabase::GetCompilationInfoForFile( const std::string &path_to_file ) { + ReleaseGil unlock; CompilationInfoForFile info; if ( !is_loaded_ ) return info; - // TODO: mutex protect calls to getCompileCommands and getDirectory + lock_guard< mutex > lock( compilation_database_mutex_ ); CompileCommandsWrap commands( clang_CompilationDatabase_getCompileCommands( @@ -120,34 +105,5 @@ CompilationInfoForFile CompilationDatabase::GetCompilationInfoForFile( return info; } - -Future< AsyncCompilationInfoForFile > -CompilationDatabase::GetCompilationInfoForFileAsync( - const std::string &path_to_file ) { - // TODO: throw exception when threading is not enabled and this is called - if ( !threading_enabled_ ) - return Future< AsyncCompilationInfoForFile >(); - - function< CompilationInfoForFile() > functor = - boost::bind( &CompilationDatabase::GetCompilationInfoForFile, - boost::ref( *this ), - path_to_file ); - - InfoTask task = - make_shared< packaged_task< AsyncCompilationInfoForFile > >( - bind( ReturnValueAsShared< CompilationInfoForFile >, - functor ) ); - - unique_future< AsyncCompilationInfoForFile > future = task->get_future(); - info_task_stack_.Push( task ); - return Future< AsyncCompilationInfoForFile >( boost::move( future ) ); -} - - -void CompilationDatabase::InitThreads() { - info_thread_ = boost::thread( QueryThreadMain, - boost::ref( info_task_stack_ ) ); -} - } // namespace YouCompleteMe diff --git a/cpp/ycm/ClangCompleter/CompilationDatabase.h b/cpp/ycm/ClangCompleter/CompilationDatabase.h index 1205f9ce..bf75b3ee 100644 --- a/cpp/ycm/ClangCompleter/CompilationDatabase.h +++ b/cpp/ycm/ClangCompleter/CompilationDatabase.h @@ -18,15 +18,14 @@ #ifndef COMPILATIONDATABASE_H_ZT7MQXPG #define COMPILATIONDATABASE_H_ZT7MQXPG -#include "Future.h" -#include "ConcurrentStack.h" - #include #include #include #include +#include #include + namespace YouCompleteMe { struct CompilationInfoForFile { @@ -34,9 +33,8 @@ struct CompilationInfoForFile { std::string compiler_working_dir_; }; -typedef boost::shared_ptr< CompilationInfoForFile > -AsyncCompilationInfoForFile; +// Access to Clang's internal CompilationDatabase. This class is thread-safe. class CompilationDatabase : boost::noncopyable { public: CompilationDatabase( const std::string &path_to_directory ); @@ -44,28 +42,20 @@ public: bool DatabaseSuccessfullyLoaded(); - void EnableThreading(); + // Returns true when a separate thread is already getting flags; this is + // useful so that the caller doesn't need to block. + bool AlreadyGettingFlags(); + // NOTE: Multiple calls to this function from separate threads will be + // serialized since Clang internals are not thread-safe. CompilationInfoForFile GetCompilationInfoForFile( const std::string &path_to_file ); - Future< AsyncCompilationInfoForFile > GetCompilationInfoForFileAsync( - const std::string &path_to_file ); - - typedef boost::shared_ptr < - boost::packaged_task< AsyncCompilationInfoForFile > > InfoTask; - - typedef ConcurrentStack< InfoTask > InfoTaskStack; - private: - void InitThreads(); - bool threading_enabled_; bool is_loaded_; CXCompilationDatabase compilation_database_; - - boost::thread info_thread_; - InfoTaskStack info_task_stack_; + boost::mutex compilation_database_mutex_; }; } // namespace YouCompleteMe diff --git a/cpp/ycm/ClangCompleter/TranslationUnit.cpp b/cpp/ycm/ClangCompleter/TranslationUnit.cpp index 0acc0681..fd01fcc6 100644 --- a/cpp/ycm/ClangCompleter/TranslationUnit.cpp +++ b/cpp/ycm/ClangCompleter/TranslationUnit.cpp @@ -57,7 +57,7 @@ TranslationUnit::TranslationUnit( std::vector< CXUnsavedFile > cxunsaved_files = ToCXUnsavedFiles( unsaved_files ); const CXUnsavedFile *unsaved = cxunsaved_files.size() > 0 - ? &cxunsaved_files [0] : NULL; + ? &cxunsaved_files[ 0 ] : NULL; clang_translation_unit_ = clang_parseTranslationUnit( clang_index, @@ -92,25 +92,11 @@ void TranslationUnit::Destroy() { std::vector< Diagnostic > TranslationUnit::LatestDiagnostics() { - std::vector< Diagnostic > diagnostics; - if ( !clang_translation_unit_ ) - return diagnostics; + return std::vector< Diagnostic >(); unique_lock< mutex > lock( diagnostics_mutex_ ); - - // We don't need the latest diags after we return them once so we swap the - // internal data with a new, empty diag vector. This vector is then returned - // and on C++11 compilers a move ctor is invoked, thus no copy is created. - // Theoretically, just returning the value of a - // [boost::|std::]move(latest_diagnostics_) call _should_ leave the - // latest_diagnostics_ vector in an emtpy, valid state but I'm not going to - // rely on that. I just had to look this up in the standard to be sure, and - // future readers of this code (myself included) should not be forced to do - // that to understand what the hell is going on. - - std::swap( latest_diagnostics_, diagnostics ); - return diagnostics; + return latest_diagnostics_; } @@ -125,12 +111,15 @@ bool TranslationUnit::IsCurrentlyUpdating() const { } -void TranslationUnit::Reparse( +std::vector< Diagnostic > TranslationUnit::Reparse( const std::vector< UnsavedFile > &unsaved_files ) { std::vector< CXUnsavedFile > cxunsaved_files = ToCXUnsavedFiles( unsaved_files ); Reparse( cxunsaved_files ); + + unique_lock< mutex > lock( diagnostics_mutex_ ); + return latest_diagnostics_; } @@ -157,7 +146,8 @@ std::vector< CompletionData > TranslationUnit::CandidatesForLocation( std::vector< CXUnsavedFile > cxunsaved_files = ToCXUnsavedFiles( unsaved_files ); const CXUnsavedFile *unsaved = cxunsaved_files.size() > 0 - ? &cxunsaved_files [0] : NULL; + ? &cxunsaved_files[ 0 ] : NULL; + // codeCompleteAt reparses the TU if the underlying source file has changed on // disk since the last time the TU was updated and there are no unsaved files. // If there are unsaved files, then codeCompleteAt will parse the in-memory @@ -253,7 +243,8 @@ void TranslationUnit::Reparse( std::vector< CXUnsavedFile > &unsaved_files, if ( !clang_translation_unit_ ) return; CXUnsavedFile *unsaved = unsaved_files.size() > 0 - ? &unsaved_files [0] : NULL; + ? &unsaved_files[ 0 ] : NULL; + failure = clang_reparseTranslationUnit( clang_translation_unit_, unsaved_files.size(), unsaved, diff --git a/cpp/ycm/ClangCompleter/TranslationUnit.h b/cpp/ycm/ClangCompleter/TranslationUnit.h index 53baf989..d1469805 100644 --- a/cpp/ycm/ClangCompleter/TranslationUnit.h +++ b/cpp/ycm/ClangCompleter/TranslationUnit.h @@ -18,8 +18,6 @@ #ifndef TRANSLATIONUNIT_H_XQ7I6SVA #define TRANSLATIONUNIT_H_XQ7I6SVA -#include "ConcurrentLatestValue.h" -#include "Future.h" #include "UnsavedFile.h" #include "Diagnostic.h" #include "Location.h" @@ -58,7 +56,8 @@ public: bool IsCurrentlyUpdating() const; - void Reparse( const std::vector< UnsavedFile > &unsaved_files ); + std::vector< Diagnostic > Reparse( + const std::vector< UnsavedFile > &unsaved_files ); void ReparseForIndexing( const std::vector< UnsavedFile > &unsaved_files ); diff --git a/cpp/ycm/ClangCompleter/TranslationUnitStore.cpp b/cpp/ycm/ClangCompleter/TranslationUnitStore.cpp index 26320024..a93d5e7e 100644 --- a/cpp/ycm/ClangCompleter/TranslationUnitStore.cpp +++ b/cpp/ycm/ClangCompleter/TranslationUnitStore.cpp @@ -21,6 +21,7 @@ #include "exceptions.h" #include +#include #include using boost::lock_guard; @@ -38,14 +39,17 @@ std::size_t HashForFlags( const std::vector< std::string > &flags ) { } // unnamed namespace + TranslationUnitStore::TranslationUnitStore( CXIndex clang_index ) : clang_index_( clang_index ) { } + TranslationUnitStore::~TranslationUnitStore() { RemoveAll(); } + shared_ptr< TranslationUnit > TranslationUnitStore::GetOrCreate( const std::string &filename, const std::vector< UnsavedFile > &unsaved_files, @@ -104,24 +108,28 @@ shared_ptr< TranslationUnit > TranslationUnitStore::GetOrCreate( return unit; } + shared_ptr< TranslationUnit > TranslationUnitStore::Get( const std::string &filename ) { lock_guard< mutex > lock( filename_to_translation_unit_and_flags_mutex_ ); return GetNoLock( filename ); } + bool TranslationUnitStore::Remove( const std::string &filename ) { lock_guard< mutex > lock( filename_to_translation_unit_and_flags_mutex_ ); Erase( filename_to_flags_hash_, filename ); return Erase( filename_to_translation_unit_, filename ); } + void TranslationUnitStore::RemoveAll() { lock_guard< mutex > lock( filename_to_translation_unit_and_flags_mutex_ ); filename_to_translation_unit_.clear(); filename_to_flags_hash_.clear(); } + shared_ptr< TranslationUnit > TranslationUnitStore::GetNoLock( const std::string &filename ) { return FindWithDefault( filename_to_translation_unit_, diff --git a/cpp/ycm/ClangCompleter/UnsavedFile.h b/cpp/ycm/ClangCompleter/UnsavedFile.h index d35041a3..69bcd2ce 100644 --- a/cpp/ycm/ClangCompleter/UnsavedFile.h +++ b/cpp/ycm/ClangCompleter/UnsavedFile.h @@ -18,13 +18,13 @@ #ifndef UNSAVEDFILE_H_0GIYZQL4 #define UNSAVEDFILE_H_0GIYZQL4 -#include +#include struct UnsavedFile { - UnsavedFile() : filename_( NULL ), contents_( NULL ), length_( 0 ) {} + UnsavedFile() : filename_( "" ), contents_( "" ), length_( 0 ) {} - const char *filename_; - const char *contents_; + std::string filename_; + std::string contents_; unsigned long length_; // We need this to be able to export this struct to Python via Boost.Python's diff --git a/cpp/ycm/ConcurrentLatestValue.h b/cpp/ycm/ConcurrentLatestValue.h deleted file mode 100644 index 97a63d76..00000000 --- a/cpp/ycm/ConcurrentLatestValue.h +++ /dev/null @@ -1,79 +0,0 @@ -// Copyright (C) 2011, 2012 Strahinja Val Markovic -// -// This file is part of YouCompleteMe. -// -// YouCompleteMe is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// YouCompleteMe is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with YouCompleteMe. If not, see . - -#ifndef CONCURRENTLATESTVALUE_H_SYF1JPPG -#define CONCURRENTLATESTVALUE_H_SYF1JPPG - -#include -#include - -namespace YouCompleteMe { - -// This is is basically a multi-consumer single-producer queue, only with the -// twist that we only care about the latest value set. So the GUI thread is the -// setter, and the worker threads are the workers. The workers wait in line on -// the condition variable and when the setter sets a value, a worker is chosen -// to consume it. -// -// The point is that we always want to have one "fresh" worker thread ready to -// work on our latest value. If a newer value is set, then we don't care what -// happens to the old values. -// -// This implementation is mutex-based and is not lock-free. Normally using a -// lock-free data structure makes more sense, but since the GUI thread goes -// through VimL and Python on every keystroke, there's really no point. Those 5 -// nanoseconds it takes to lock a mutex are laughably negligible compared to the -// VimL/Python overhead. -template -class ConcurrentLatestValue : boost::noncopyable { -public: - - ConcurrentLatestValue() : empty_( true ) {} - - void Set( const T &data ) { - { - boost::unique_lock< boost::mutex > lock( mutex_ ); - latest_ = data; - empty_ = false; - } - - condition_variable_.notify_one(); - } - - T Get() { - boost::unique_lock< boost::mutex > lock( mutex_ ); - - while ( empty_ ) { - condition_variable_.wait( lock ); - } - - empty_ = true; - return latest_; - } - - -private: - T latest_; - bool empty_; - boost::mutex mutex_; - boost::condition_variable condition_variable_; - -}; - -} // namespace YouCompleteMe - -#endif /* end of include guard: CONCURRENTLATESTVALUE_H_SYF1JPPG */ diff --git a/cpp/ycm/ConcurrentStack.h b/cpp/ycm/ConcurrentStack.h deleted file mode 100644 index 5d88bd0a..00000000 --- a/cpp/ycm/ConcurrentStack.h +++ /dev/null @@ -1,89 +0,0 @@ -// Copyright (C) 2011, 2012 Strahinja Val Markovic -// -// This file is part of YouCompleteMe. -// -// YouCompleteMe is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// YouCompleteMe is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with YouCompleteMe. If not, see . - -#ifndef CONCURRENTSTACK_H_TGI0GOR6 -#define CONCURRENTSTACK_H_TGI0GOR6 - -#include -#include -#include - -namespace YouCompleteMe { - -template -class ConcurrentStack : boost::noncopyable { -public: - - void Push( const T &data ) { - { - boost::unique_lock< boost::mutex > lock( mutex_ ); - stack_.push( data ); - } - - condition_variable_.notify_one(); - } - - - T Pop() { - boost::unique_lock< boost::mutex > lock( mutex_ ); - - while ( stack_.empty() ) { - condition_variable_.wait( lock ); - } - - T top = stack_.top(); - stack_.pop(); - return top; - } - - - // Gets all the items from the stack and appends them to the input vector. - // Does not wait for the stack to get items; if the stack is empty, returns - // false and does not touch the input vector. - bool PopAllNoWait( std::vector< T > &items ) { - boost::unique_lock< boost::mutex > lock( mutex_ ); - - if ( stack_.empty() ) - return false; - - int num_items = stack_.size(); - items.reserve( num_items + items.size() ); - - for ( int i = 0; i < num_items; ++i ) { - items.push_back( stack_.top() ); - stack_.pop(); - } - - return true; - } - - - bool Empty() { - boost::unique_lock< boost::mutex > lock( mutex_ ); - return stack_.empty(); - } - -private: - std::stack< T > stack_; - boost::mutex mutex_; - boost::condition_variable condition_variable_; - -}; - -} // namespace YouCompleteMe - -#endif /* end of include guard: CONCURRENTSTACK_H_TGI0GOR6 */ diff --git a/cpp/ycm/Future.h b/cpp/ycm/Future.h deleted file mode 100644 index 235eea24..00000000 --- a/cpp/ycm/Future.h +++ /dev/null @@ -1,78 +0,0 @@ -// Copyright (C) 2011, 2012 Strahinja Val Markovic -// -// This file is part of YouCompleteMe. -// -// YouCompleteMe is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// YouCompleteMe is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with YouCompleteMe. If not, see . - -#ifndef FUTURE_H_NR1U6MZS -#define FUTURE_H_NR1U6MZS - -#include -#include -#include -#include - -namespace YouCompleteMe { - -class Result; -typedef boost::shared_ptr< boost::packaged_task< void > > VoidTask; - -template< typename T > -boost::shared_ptr< T > ReturnValueAsShared( - boost::function< T() > func ) { - return boost::make_shared< T >( func() ); -} - - -template< typename T > -class Future { -public: - Future() {}; - Future( boost::shared_future< T > future ) - : future_( boost::move( future ) ) {} - - bool ResultsReady() { - // It's OK to return true since GetResults will just return a - // default-constructed value if the future_ is uninitialized. If we don't - // return true for this case, any loop waiting on ResultsReady will wait - // forever. - if ( future_.get_state() == boost::future_state::uninitialized ) - return true; - - return future_.is_ready(); - } - - - void Wait() { - future_.wait(); - } - - - T GetResults() { - try { - return future_.get(); - } catch ( boost::future_uninitialized & ) { - // Do nothing and return a T() - } - - return T(); - } - -private: - boost::shared_future< T > future_; -}; - -} // namespace YouCompleteMe - -#endif /* end of include guard: FUTURE_H_NR1U6MZS */ diff --git a/cpp/ycm/IdentifierCompleter.cpp b/cpp/ycm/IdentifierCompleter.cpp index 88294ae1..a6bd2674 100644 --- a/cpp/ycm/IdentifierCompleter.cpp +++ b/cpp/ycm/IdentifierCompleter.cpp @@ -22,63 +22,18 @@ #include "IdentifierUtils.h" #include "Result.h" #include "Utils.h" +#include "ReleaseGil.h" -#include -#include #include -using boost::packaged_task; -using boost::unique_future; -using boost::shared_ptr; -using boost::thread; - namespace YouCompleteMe { -typedef boost::function< std::vector< std::string >() > -FunctionReturnsStringVector; - -extern const unsigned int MAX_ASYNC_THREADS = 4; -extern const unsigned int MIN_ASYNC_THREADS = 2; - -namespace { - -void QueryThreadMain( - IdentifierCompleter::LatestQueryTask &latest_query_task ) { - while ( true ) { - try { - ( *latest_query_task.Get() )(); - } catch ( boost::thread_interrupted & ) { - return; - } - } - -} - -void BufferIdentifiersThreadMain( - IdentifierCompleter::BufferIdentifiersTaskStack - &buffer_identifiers_task_stack ) { - while ( true ) { - try { - ( *buffer_identifiers_task_stack.Pop() )(); - } catch ( boost::thread_interrupted & ) { - return; - } - } -} - - -} // unnamed namespace - - -IdentifierCompleter::IdentifierCompleter() - : threading_enabled_( false ) { -} +IdentifierCompleter::IdentifierCompleter() {} IdentifierCompleter::IdentifierCompleter( - const std::vector< std::string > &candidates ) - : threading_enabled_( false ) { + const std::vector< std::string > &candidates ) { identifier_database_.AddIdentifiers( candidates, "", "" ); } @@ -86,35 +41,16 @@ IdentifierCompleter::IdentifierCompleter( IdentifierCompleter::IdentifierCompleter( const std::vector< std::string > &candidates, const std::string &filetype, - const std::string &filepath ) - : threading_enabled_( false ) { + const std::string &filepath ) { identifier_database_.AddIdentifiers( candidates, filetype, filepath ); } -IdentifierCompleter::~IdentifierCompleter() { - query_threads_.interrupt_all(); - query_threads_.join_all(); - - if ( buffer_identifiers_thread_ ) { - buffer_identifiers_thread_->interrupt(); - buffer_identifiers_thread_->join(); - } -} - - -// We need this mostly so that we can not use it in tests. Apparently the -// GoogleTest framework goes apeshit on us if we enable threads by default. -void IdentifierCompleter::EnableThreading() { - threading_enabled_ = true; - InitThreads(); -} - - void IdentifierCompleter::AddIdentifiersToDatabase( const std::vector< std::string > &new_candidates, const std::string &filetype, const std::string &filepath ) { + ReleaseGil unlock; identifier_database_.AddIdentifiers( new_candidates, filetype, filepath ); @@ -123,6 +59,7 @@ void IdentifierCompleter::AddIdentifiersToDatabase( void IdentifierCompleter::AddIdentifiersToDatabaseFromTagFiles( const std::vector< std::string > &absolute_paths_to_tag_files ) { + ReleaseGil unlock; foreach( const std::string & path, absolute_paths_to_tag_files ) { identifier_database_.AddIdentifiers( ExtractIdentifiersFromTagsFile( path ) ); @@ -130,27 +67,12 @@ void IdentifierCompleter::AddIdentifiersToDatabaseFromTagFiles( } -void IdentifierCompleter::AddIdentifiersToDatabaseFromTagFilesAsync( - std::vector< std::string > absolute_paths_to_tag_files ) { - // TODO: throw exception when threading is not enabled and this is called - if ( !threading_enabled_ ) - return; - - boost::function< void() > functor = - boost::bind( &IdentifierCompleter::AddIdentifiersToDatabaseFromTagFiles, - boost::ref( *this ), - boost::move( absolute_paths_to_tag_files ) ); - - buffer_identifiers_task_stack_.Push( - boost::make_shared< packaged_task< void > >( boost::move( functor ) ) ); -} - - void IdentifierCompleter::AddIdentifiersToDatabaseFromBuffer( const std::string &buffer_contents, const std::string &filetype, const std::string &filepath, bool collect_from_comments_and_strings ) { + ReleaseGil unlock; identifier_database_.ClearCandidatesStoredForFile( filetype, filepath ); std::string new_contents = @@ -165,28 +87,6 @@ void IdentifierCompleter::AddIdentifiersToDatabaseFromBuffer( } -void IdentifierCompleter::AddIdentifiersToDatabaseFromBufferAsync( - std::string buffer_contents, - std::string filetype, - std::string filepath, - bool collect_from_comments_and_strings ) { - // TODO: throw exception when threading is not enabled and this is called - if ( !threading_enabled_ ) - return; - - boost::function< void() > functor = - boost::bind( &IdentifierCompleter::AddIdentifiersToDatabaseFromBuffer, - boost::ref( *this ), - boost::move( buffer_contents ), - boost::move( filetype ), - boost::move( filepath ), - collect_from_comments_and_strings ); - - buffer_identifiers_task_stack_.Push( - boost::make_shared< packaged_task< void > >( boost::move( functor ) ) ); -} - - std::vector< std::string > IdentifierCompleter::CandidatesForQuery( const std::string &query ) const { return CandidatesForQueryAndType( query, "" ); @@ -196,6 +96,7 @@ std::vector< std::string > IdentifierCompleter::CandidatesForQuery( std::vector< std::string > IdentifierCompleter::CandidatesForQueryAndType( const std::string &query, const std::string &filetype ) const { + ReleaseGil unlock; std::vector< Result > results; identifier_database_.ResultsForQueryAndType( query, filetype, results ); @@ -209,46 +110,4 @@ std::vector< std::string > IdentifierCompleter::CandidatesForQueryAndType( } -Future< AsyncResults > IdentifierCompleter::CandidatesForQueryAndTypeAsync( - const std::string &query, - const std::string &filetype ) const { - // TODO: throw exception when threading is not enabled and this is called - if ( !threading_enabled_ ) - return Future< AsyncResults >(); - - FunctionReturnsStringVector functor = - boost::bind( &IdentifierCompleter::CandidatesForQueryAndType, - boost::cref( *this ), - query, - filetype ); - - QueryTask task = - boost::make_shared< packaged_task< AsyncResults > >( - boost::bind( ReturnValueAsShared< std::vector< std::string > >, - boost::move( functor ) ) ); - - unique_future< AsyncResults > future = task->get_future(); - - latest_query_task_.Set( task ); - return Future< AsyncResults >( boost::move( future ) ); -} - - -void IdentifierCompleter::InitThreads() { - int query_threads_to_create = - std::max( MIN_ASYNC_THREADS, - std::min( MAX_ASYNC_THREADS, thread::hardware_concurrency() ) ); - - for ( int i = 0; i < query_threads_to_create; ++i ) { - query_threads_.create_thread( - boost::bind( QueryThreadMain, - boost::ref( latest_query_task_ ) ) ); - } - - buffer_identifiers_thread_.reset( - new boost::thread( BufferIdentifiersThreadMain, - boost::ref( buffer_identifiers_task_stack_ ) ) ); -} - - } // namespace YouCompleteMe diff --git a/cpp/ycm/IdentifierCompleter.h b/cpp/ycm/IdentifierCompleter.h index bc7dd5fd..5b14420b 100644 --- a/cpp/ycm/IdentifierCompleter.h +++ b/cpp/ycm/IdentifierCompleter.h @@ -18,10 +18,7 @@ #ifndef COMPLETER_H_7AR4UGXE #define COMPLETER_H_7AR4UGXE -#include "ConcurrentLatestValue.h" -#include "ConcurrentStack.h" #include "IdentifierDatabase.h" -#include "Future.h" #include #include @@ -36,8 +33,6 @@ namespace YouCompleteMe { class Candidate; -typedef boost::shared_ptr< std::vector< std::string > > AsyncResults; - class IdentifierCompleter : boost::noncopyable { public: @@ -47,10 +42,6 @@ public: const std::string &filetype, const std::string &filepath ); - ~IdentifierCompleter(); - - void EnableThreading(); - void AddIdentifiersToDatabase( const std::vector< std::string > &new_candidates, const std::string &filetype, @@ -59,25 +50,12 @@ public: void AddIdentifiersToDatabaseFromTagFiles( const std::vector< std::string > &absolute_paths_to_tag_files ); - // NOTE: params are taken by value on purpose! - void AddIdentifiersToDatabaseFromTagFilesAsync( - std::vector< std::string > absolute_paths_to_tag_files ); - void AddIdentifiersToDatabaseFromBuffer( const std::string &buffer_contents, const std::string &filetype, const std::string &filepath, bool collect_from_comments_and_strings ); - // NOTE: params are taken by value on purpose! With a C++11 compiler we can - // avoid an expensive copy of buffer_contents if the param is taken by value - // (move ctors FTW) - void AddIdentifiersToDatabaseFromBufferAsync( - std::string buffer_contents, - std::string filetype, - std::string filepath, - bool collect_from_comments_and_strings ); - // Only provided for tests! std::vector< std::string > CandidatesForQuery( const std::string &query ) const; @@ -86,37 +64,13 @@ public: const std::string &query, const std::string &filetype ) const; - Future< AsyncResults > CandidatesForQueryAndTypeAsync( - const std::string &query, - const std::string &filetype ) const; - - typedef boost::shared_ptr < - boost::packaged_task< AsyncResults > > QueryTask; - - typedef ConcurrentLatestValue< QueryTask > LatestQueryTask; - - typedef ConcurrentStack< VoidTask > BufferIdentifiersTaskStack; - private: - void InitThreads(); - - ///////////////////////////// // PRIVATE MEMBER VARIABLES ///////////////////////////// IdentifierDatabase identifier_database_; - - bool threading_enabled_; - - boost::thread_group query_threads_; - - boost::scoped_ptr< boost::thread > buffer_identifiers_thread_; - - mutable LatestQueryTask latest_query_task_; - - BufferIdentifiersTaskStack buffer_identifiers_task_stack_; }; } // namespace YouCompleteMe diff --git a/cpp/ycm/PythonSupport.cpp b/cpp/ycm/PythonSupport.cpp index 635ae5ed..1b14cd63 100644 --- a/cpp/ycm/PythonSupport.cpp +++ b/cpp/ycm/PythonSupport.cpp @@ -20,6 +20,8 @@ #include "Result.h" #include "Candidate.h" #include "CandidateRepository.h" +#include "ReleaseGil.h" + #include #include #include @@ -27,6 +29,7 @@ using boost::algorithm::any_of; using boost::algorithm::is_upper; using boost::python::len; +using boost::python::str; using boost::python::extract; using boost::python::object; typedef boost::python::list pylist; @@ -35,6 +38,13 @@ namespace YouCompleteMe { namespace { +std::string GetUtf8String( const boost::python::object &string_or_unicode ) { + extract< std::string > to_string( string_or_unicode ); + if ( to_string.check() ) + return to_string(); + return extract< std::string >( str( string_or_unicode ).encode( "utf8" ) ); +} + std::vector< const Candidate * > CandidatesFromObjectList( const pylist &candidates, const std::string &candidate_property ) { @@ -44,10 +54,10 @@ std::vector< const Candidate * > CandidatesFromObjectList( for ( int i = 0; i < num_candidates; ++i ) { if ( candidate_property.empty() ) { - candidate_strings.push_back( extract< std::string >( candidates[ i ] ) ); + candidate_strings.push_back( GetUtf8String( candidates[ i ] ) ); } else { object holder = extract< object >( candidates[ i ] ); - candidate_strings.push_back( extract< std::string >( + candidate_strings.push_back( GetUtf8String( holder[ candidate_property.c_str() ] ) ); } } @@ -68,31 +78,33 @@ boost::python::list FilterAndSortCandidates( return candidates; } + int num_candidates = len( candidates ); std::vector< const Candidate * > repository_candidates = CandidatesFromObjectList( candidates, candidate_property ); - Bitset query_bitset = LetterBitsetFromString( query ); - bool query_has_uppercase_letters = any_of( query, is_upper() ); - - int num_candidates = len( candidates ); std::vector< ResultAnd< int > > object_and_results; + { + ReleaseGil unlock; + Bitset query_bitset = LetterBitsetFromString( query ); + bool query_has_uppercase_letters = any_of( query, is_upper() ); - for ( int i = 0; i < num_candidates; ++i ) { - const Candidate *candidate = repository_candidates[ i ]; + for ( int i = 0; i < num_candidates; ++i ) { + const Candidate *candidate = repository_candidates[ i ]; - if ( !candidate->MatchesQueryBitset( query_bitset ) ) - continue; + if ( !candidate->MatchesQueryBitset( query_bitset ) ) + continue; - Result result = candidate->QueryMatchResult( query, - query_has_uppercase_letters ); + Result result = candidate->QueryMatchResult( query, + query_has_uppercase_letters ); - if ( result.IsSubsequence() ) { - ResultAnd< int > object_and_result( i, result ); - object_and_results.push_back( boost::move( object_and_result ) ); + if ( result.IsSubsequence() ) { + ResultAnd< int > object_and_result( i, result ); + object_and_results.push_back( boost::move( object_and_result ) ); + } } - } - std::sort( object_and_results.begin(), object_and_results.end() ); + std::sort( object_and_results.begin(), object_and_results.end() ); + } foreach ( const ResultAnd< int > &object_and_result, object_and_results ) { diff --git a/cpp/ycm/ClangCompleter/ClangResultsCache.cpp b/cpp/ycm/ReleaseGil.h similarity index 50% rename from cpp/ycm/ClangCompleter/ClangResultsCache.cpp rename to cpp/ycm/ReleaseGil.h index e11856e3..c22de2be 100644 --- a/cpp/ycm/ClangCompleter/ClangResultsCache.cpp +++ b/cpp/ycm/ReleaseGil.h @@ -1,4 +1,4 @@ -// Copyright (C) 2011, 2012 Strahinja Val Markovic +// Copyright (C) 2013 Strahinja Val Markovic // // This file is part of YouCompleteMe. // @@ -15,29 +15,28 @@ // You should have received a copy of the GNU General Public License // along with YouCompleteMe. If not, see . -#include "ClangResultsCache.h" -#include "standard.h" +#ifndef RELEASEGIL_H_RDIEBSQ1 +#define RELEASEGIL_H_RDIEBSQ1 -using boost::shared_mutex; -using boost::shared_lock; -using boost::unique_lock; +#include namespace YouCompleteMe { -bool ClangResultsCache::NewPositionDifferentFromStoredPosition( int new_line, - int new_colum ) -const { - shared_lock< shared_mutex > reader_lock( access_mutex_ ); - return line_ != new_line || column_ != new_colum; -} +class ReleaseGil { +public: + ReleaseGil() { + thread_state_ = PyEval_SaveThread(); + } -void ClangResultsCache::ResetWithNewLineAndColumn( int new_line, - int new_colum ) { - unique_lock< shared_mutex > reader_lock( access_mutex_ ); + ~ReleaseGil() { + PyEval_RestoreThread( thread_state_ ); + } - line_ = new_line; - column_ = new_colum; - completion_datas_.clear(); -} +private: + PyThreadState *thread_state_; +}; } // namespace YouCompleteMe + +#endif /* end of include guard: RELEASEGIL_H_RDIEBSQ1 */ + diff --git a/cpp/ycm/Result.h b/cpp/ycm/Result.h index a9c6cab4..2aaf8c4d 100644 --- a/cpp/ycm/Result.h +++ b/cpp/ycm/Result.h @@ -88,6 +88,7 @@ private: template< class T > struct ResultAnd { + // TODO: Swap the order of these parameters ResultAnd( T extra_object, const Result &result ) : extra_object_( extra_object ), result_( result ) {} diff --git a/cpp/ycm/tests/CMakeLists.txt b/cpp/ycm/tests/CMakeLists.txt index c4d8aa7c..1383cc0f 100644 --- a/cpp/ycm/tests/CMakeLists.txt +++ b/cpp/ycm/tests/CMakeLists.txt @@ -28,7 +28,7 @@ endif() add_subdirectory( gmock ) include_directories( - ${ycm_core_SOURCE_DIR} + ${ycm_support_libs_SOURCE_DIR} ) include_directories( @@ -67,8 +67,9 @@ add_executable( ${PROJECT_NAME} ) target_link_libraries( ${PROJECT_NAME} - ycm_core - gmock_main ) + ${SERVER_LIB} + ${CLIENT_LIB} + gmock ) if ( NOT CMAKE_GENERATOR_IS_XCODE ) diff --git a/cpp/ycm/tests/ClangCompleter/ClangCompleter_test.cpp b/cpp/ycm/tests/ClangCompleter/ClangCompleter_test.cpp index c40aa1ee..18be50d2 100644 --- a/cpp/ycm/tests/ClangCompleter/ClangCompleter_test.cpp +++ b/cpp/ycm/tests/ClangCompleter/ClangCompleter_test.cpp @@ -43,22 +43,21 @@ TEST( ClangCompleterTest, CandidatesForLocationInFile ) { } -TEST( ClangCompleterTest, CandidatesForQueryAndLocationInFileAsync ) { +TEST( ClangCompleterTest, GetDefinitionLocation ) { ClangCompleter completer; - completer.EnableThreading(); + std::string filename = PathToTestFile( "basic.cpp" ).string(); - Future< AsyncCompletions > completions_future = - completer.CandidatesForQueryAndLocationInFileAsync( - "", - PathToTestFile( "basic.cpp" ).string(), - 11, - 7, + // Clang operates on the reasonable assumption that line and column numbers + // are 1-based. + Location actual_location = + completer.GetDefinitionLocation( + filename, + 9, + 3, std::vector< UnsavedFile >(), std::vector< std::string >() ); - completions_future.Wait(); - - EXPECT_TRUE( !completions_future.GetResults()->empty() ); + EXPECT_EQ( Location( filename, 1, 8 ), actual_location ); } } // namespace YouCompleteMe diff --git a/cpp/ycm/tests/IdentifierCompleter_test.cpp b/cpp/ycm/tests/IdentifierCompleter_test.cpp index 959b8f53..d3c4d0dc 100644 --- a/cpp/ycm/tests/IdentifierCompleter_test.cpp +++ b/cpp/ycm/tests/IdentifierCompleter_test.cpp @@ -217,6 +217,18 @@ TEST( IdentifierCompleterTest, ShorterAndLowercaseWins ) { "STDIN_FILENO" ) ); } +TEST( IdentifierCompleterTest, AddIdentifiersToDatabaseFromBufferWorks ) { + IdentifierCompleter completer; + completer.AddIdentifiersToDatabaseFromBuffer( "foo foogoo ba", + "foo", + "/foo/bar", + false ); + + EXPECT_THAT( completer.CandidatesForQueryAndType( "oo", "foo" ), + ElementsAre( "foo", + "foogoo" ) ); +} + TEST( IdentifierCompleterTest, TagsEndToEndWorks ) { IdentifierCompleter completer; std::vector< std::string > tag_files; diff --git a/cpp/ycm/tests/main.cpp b/cpp/ycm/tests/main.cpp new file mode 100644 index 00000000..62c5198c --- /dev/null +++ b/cpp/ycm/tests/main.cpp @@ -0,0 +1,13 @@ +#include "gtest/gtest.h" +#include "gmock/gmock.h" +#include + +int main( int argc, char **argv ) { + Py_Initialize(); + // Necessary because of usage of the ReleaseGil class + PyEval_InitThreads(); + + testing::InitGoogleMock(&argc, argv); + return RUN_ALL_TESTS(); +} + diff --git a/cpp/ycm/tests/testdata/basic.cpp b/cpp/ycm/tests/testdata/basic.cpp index b5bdf7f6..722d6bc3 100644 --- a/cpp/ycm/tests/testdata/basic.cpp +++ b/cpp/ycm/tests/testdata/basic.cpp @@ -2,7 +2,7 @@ struct Foo { int x; int y; char c; -} +}; int main() { diff --git a/cpp/ycm/versioning.cpp b/cpp/ycm/versioning.cpp new file mode 100644 index 00000000..0007ff2e --- /dev/null +++ b/cpp/ycm/versioning.cpp @@ -0,0 +1,26 @@ +// Copyright (C) 2013 Strahinja Val Markovic +// +// This file is part of YouCompleteMe. +// +// YouCompleteMe is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// YouCompleteMe is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with YouCompleteMe. If not, see . + +namespace YouCompleteMe { + +int YcmCoreVersion() { + // We increment this every time when we want to force users to recompile + // ycm_core. + return 7; +} + +} // namespace YouCompleteMe diff --git a/cpp/ycm/versioning.h b/cpp/ycm/versioning.h new file mode 100644 index 00000000..2b2fc081 --- /dev/null +++ b/cpp/ycm/versioning.h @@ -0,0 +1,22 @@ +// Copyright (C) 2013 Strahinja Val Markovic +// +// This file is part of YouCompleteMe. +// +// YouCompleteMe is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// YouCompleteMe is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with YouCompleteMe. If not, see . + +namespace YouCompleteMe { + +int YcmCoreVersion(); + +} // namespace YouCompleteMe diff --git a/cpp/ycm/ycm_client_support.cpp b/cpp/ycm/ycm_client_support.cpp new file mode 100644 index 00000000..25b77194 --- /dev/null +++ b/cpp/ycm/ycm_client_support.cpp @@ -0,0 +1,44 @@ +// Copyright (C) 2013 Strahinja Val Markovic +// +// This file is part of YouCompleteMe. +// +// YouCompleteMe is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// YouCompleteMe is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with YouCompleteMe. If not, see . + +#include "IdentifierCompleter.h" +#include "PythonSupport.h" +#include "versioning.h" + +#include +#include + + +BOOST_PYTHON_MODULE(ycm_client_support) +{ + using namespace boost::python; + using namespace YouCompleteMe; + + // Necessary because of usage of the ReleaseGil class + PyEval_InitThreads(); + + def( "FilterAndSortCandidates", FilterAndSortCandidates ); + def( "YcmCoreVersion", YcmCoreVersion ); +} + +// Boost.Thread forces us to implement this. +// We don't use any thread-specific (local) storage so it's fine to implement +// this as an empty function. +namespace boost { +void tss_cleanup_implemented() {} +}; + diff --git a/cpp/ycm/ycm_core.cpp b/cpp/ycm/ycm_core.cpp index 4c1ba0d9..be347652 100644 --- a/cpp/ycm/ycm_core.cpp +++ b/cpp/ycm/ycm_core.cpp @@ -17,7 +17,7 @@ #include "IdentifierCompleter.h" #include "PythonSupport.h" -#include "Future.h" +#include "versioning.h" #ifdef USE_CLANG_COMPLETER # include "ClangCompleter.h" @@ -33,8 +33,7 @@ #include #include -bool HasClangSupport() -{ +bool HasClangSupport() { #ifdef USE_CLANG_COMPLETER return true; #else @@ -42,61 +41,37 @@ bool HasClangSupport() #endif // USE_CLANG_COMPLETER } -int YcmCoreVersion() -{ - // We increment this every time when we want to force users to recompile - // ycm_core. - return 4; -} - BOOST_PYTHON_MODULE(ycm_core) { using namespace boost::python; using namespace YouCompleteMe; + // Necessary because of usage of the ReleaseGil class + PyEval_InitThreads(); + def( "HasClangSupport", HasClangSupport ); def( "FilterAndSortCandidates", FilterAndSortCandidates ); def( "YcmCoreVersion", YcmCoreVersion ); class_< IdentifierCompleter, boost::noncopyable >( "IdentifierCompleter" ) - .def( "EnableThreading", &IdentifierCompleter::EnableThreading ) .def( "AddIdentifiersToDatabase", &IdentifierCompleter::AddIdentifiersToDatabase ) - .def( "AddIdentifiersToDatabaseFromTagFilesAsync", - &IdentifierCompleter::AddIdentifiersToDatabaseFromTagFilesAsync ) - .def( "AddIdentifiersToDatabaseFromBufferAsync", - &IdentifierCompleter::AddIdentifiersToDatabaseFromBufferAsync ) - .def( "CandidatesForQueryAndTypeAsync", - &IdentifierCompleter::CandidatesForQueryAndTypeAsync ); + .def( "AddIdentifiersToDatabaseFromTagFiles", + &IdentifierCompleter::AddIdentifiersToDatabaseFromTagFiles ) + .def( "AddIdentifiersToDatabaseFromBuffer", + &IdentifierCompleter::AddIdentifiersToDatabaseFromBuffer ) + .def( "CandidatesForQueryAndType", + &IdentifierCompleter::CandidatesForQueryAndType ); // TODO: rename these *Vec classes to *Vector; don't forget the python file class_< std::vector< std::string >, boost::shared_ptr< std::vector< std::string > > >( "StringVec" ) .def( vector_indexing_suite< std::vector< std::string > >() ); - class_< Future< AsyncResults > >( "FutureResults" ) - .def( "ResultsReady", &Future< AsyncResults >::ResultsReady ) - .def( "GetResults", &Future< AsyncResults >::GetResults ); - - class_< Future< void > >( "FutureVoid" ) - .def( "ResultsReady", &Future< void >::ResultsReady ) - .def( "GetResults", &Future< void >::GetResults ); - #ifdef USE_CLANG_COMPLETER def( "ClangVersion", ClangVersion ); - class_< Future< AsyncCompletions > >( "FutureCompletions" ) - .def( "ResultsReady", &Future< AsyncCompletions >::ResultsReady ) - .def( "GetResults", &Future< AsyncCompletions >::GetResults ); - - class_< Future< AsyncCompilationInfoForFile > >( - "FutureCompilationInfoForFile" ) - .def( "ResultsReady", - &Future< AsyncCompilationInfoForFile >::ResultsReady ) - .def( "GetResults", - &Future< AsyncCompilationInfoForFile >::GetResults ); - // CAREFUL HERE! For filename and contents we are referring directly to // Python-allocated and -managed memory since we are accepting pointers to // data members of python objects. We need to ensure that those objects @@ -116,17 +91,13 @@ BOOST_PYTHON_MODULE(ycm_core) .def( vector_indexing_suite< std::vector< UnsavedFile > >() ); class_< ClangCompleter, boost::noncopyable >( "ClangCompleter" ) - .def( "EnableThreading", &ClangCompleter::EnableThreading ) - .def( "DiagnosticsForFile", &ClangCompleter::DiagnosticsForFile ) .def( "GetDeclarationLocation", &ClangCompleter::GetDeclarationLocation ) .def( "GetDefinitionLocation", &ClangCompleter::GetDefinitionLocation ) - .def( "DeleteCachesForFileAsync", - &ClangCompleter::DeleteCachesForFileAsync ) + .def( "DeleteCachesForFile", &ClangCompleter::DeleteCachesForFile ) .def( "UpdatingTranslationUnit", &ClangCompleter::UpdatingTranslationUnit ) - .def( "UpdateTranslationUnitAsync", - &ClangCompleter::UpdateTranslationUnitAsync ) - .def( "CandidatesForQueryAndLocationInFileAsync", - &ClangCompleter::CandidatesForQueryAndLocationInFileAsync ); + .def( "UpdateTranslationUnit", &ClangCompleter::UpdateTranslationUnit ) + .def( "CandidatesForLocationInFile", + &ClangCompleter::CandidatesForLocationInFile ); class_< CompletionData >( "CompletionData" ) .def( "TextToInsertInBuffer", &CompletionData::TextToInsertInBuffer ) @@ -160,13 +131,12 @@ BOOST_PYTHON_MODULE(ycm_core) class_< CompilationDatabase, boost::noncopyable >( "CompilationDatabase", init< std::string >() ) - .def( "EnableThreading", &CompilationDatabase::EnableThreading ) .def( "DatabaseSuccessfullyLoaded", &CompilationDatabase::DatabaseSuccessfullyLoaded ) + .def( "AlreadyGettingFlags", + &CompilationDatabase::AlreadyGettingFlags ) .def( "GetCompilationInfoForFile", - &CompilationDatabase::GetCompilationInfoForFile ) - .def( "GetCompilationInfoForFileAsync", - &CompilationDatabase::GetCompilationInfoForFileAsync ); + &CompilationDatabase::GetCompilationInfoForFile ); class_< CompilationInfoForFile, boost::shared_ptr< CompilationInfoForFile > >( diff --git a/doc/youcompleteme.txt b/doc/youcompleteme.txt index ad5a4af4..50cc69c1 100644 --- a/doc/youcompleteme.txt +++ b/doc/youcompleteme.txt @@ -1,4 +1,4 @@ -*youcompleteme* YouCompleteMe: a code-completion engine for Vim +*youcompleteme.txt* YouCompleteMe: a code-completion engine for Vim =============================================================================== Contents ~ @@ -10,20 +10,22 @@ Contents ~ 5. Full Installation Guide |youcompleteme-full-installation-guide| 6. User Guide |youcompleteme-user-guide| 1. General Usage |youcompleteme-general-usage| - 2. Completion string ranking |youcompleteme-completion-string-ranking| - 3. General Semantic Completion Engine Usage |youcompleteme-general-semantic-completion-engine-usage| - 4. C-family Semantic Completion Engine Usage |youcompleteme-c-family-semantic-completion-engine-usage| - 5. Python semantic completion |youcompleteme-python-semantic-completion| - 6. C# semantic completion |youcompleteme-c-semantic-completion| - 7. Semantic completion for other languages |youcompleteme-semantic-completion-for-other-languages| - 8. Writing New Semantic Completers |youcompleteme-writing-new-semantic-completers| - 9. Syntastic integration |youcompleteme-syntastic-integration| + 2. Client-server architecture |youcompleteme-client-server-architecture| + 3. Completion string ranking |youcompleteme-completion-string-ranking| + 4. General Semantic Completion Engine Usage |youcompleteme-general-semantic-completion-engine-usage| + 5. C-family Semantic Completion Engine Usage |youcompleteme-c-family-semantic-completion-engine-usage| + 6. Python semantic completion |youcompleteme-python-semantic-completion| + 7. C# semantic completion |youcompleteme-c-semantic-completion| + 8. Semantic completion for other languages |youcompleteme-semantic-completion-for-other-languages| + 9. Writing New Semantic Completers |youcompleteme-writing-new-semantic-completers| + 10. Syntastic integration |youcompleteme-syntastic-integration| 7. Commands |youcompleteme-commands| - 1. The |:YcmForceCompileAndDiagnostics| command - 2. The |:YcmDiags| command - 3. The |:YcmShowDetailedDiagnostic| command - 4. The |:YcmDebugInfo| command - 5. The |:YcmCompleter| command + 1. The |:YcmRestartServer| command + 2. The |:YcmForceCompileAndDiagnostics| command + 3. The |:YcmDiags| command + 4. The |:YcmShowDetailedDiagnostic| command + 5. The |:YcmDebugInfo| command + 6. The |:YcmCompleter| command 8. YcmCompleter subcommands |youcompleteme-ycmcompleter-subcommands| 1. The |GoToDeclaration| subcommand 2. The |GoToDefinition| subcommand @@ -45,23 +47,27 @@ Contents ~ 10. The |g:ycm_collect_identifiers_from_comments_and_strings| option 11. The |g:ycm_collect_identifiers_from_tags_files| option 12. The |g:ycm_seed_identifiers_with_syntax| option - 13. The |g:ycm_csharp_server_port| option - 14. The |g:ycm_auto_start_csharp_server| option - 15. The |g:ycm_auto_stop_csharp_server| option - 16. The |g:ycm_add_preview_to_completeopt| option - 17. The |g:ycm_autoclose_preview_window_after_completion| option - 18. The |g:ycm_autoclose_preview_window_after_insertion| option - 19. The |g:ycm_max_diagnostics_to_display| option - 20. The |g:ycm_key_list_select_completion| option - 21. The |g:ycm_key_list_previous_completion| option - 22. The |g:ycm_key_invoke_completion| option - 23. The |g:ycm_key_detailed_diagnostics| option - 24. The |g:ycm_global_ycm_extra_conf| option - 25. The |g:ycm_confirm_extra_conf| option - 26. The |g:ycm_extra_conf_globlist| option - 27. The |g:ycm_filepath_completion_use_working_dir| option - 28. The |g:ycm_semantic_triggers| option - 29. The |g:ycm_cache_omnifunc| option + 13. The |g:ycm_server_use_vim_stdout| option + 14. The |g:ycm_server_keep_logfiles| option + 15. The |g:ycm_server_log_level| option + 16. The |g:ycm_server_idle_suicide_seconds| option + 17. The |g:ycm_csharp_server_port| option + 18. The |g:ycm_auto_start_csharp_server| option + 19. The |g:ycm_auto_stop_csharp_server| option + 20. The |g:ycm_add_preview_to_completeopt| option + 21. The |g:ycm_autoclose_preview_window_after_completion| option + 22. The |g:ycm_autoclose_preview_window_after_insertion| option + 23. The |g:ycm_max_diagnostics_to_display| option + 24. The |g:ycm_key_list_select_completion| option + 25. The |g:ycm_key_list_previous_completion| option + 26. The |g:ycm_key_invoke_completion| option + 27. The |g:ycm_key_detailed_diagnostics| option + 28. The |g:ycm_global_ycm_extra_conf| option + 29. The |g:ycm_confirm_extra_conf| option + 30. The |g:ycm_extra_conf_globlist| option + 31. The |g:ycm_filepath_completion_use_working_dir| option + 32. The |g:ycm_semantic_triggers| option + 33. The |g:ycm_cache_omnifunc| option 10. FAQ |youcompleteme-faq| 1. I get a linker warning regarding |libpython| on Mac when compiling YCM 2. I get a weird window at the top of my file when I use the semantic engine |youcompleteme-i-get-weird-window-at-top-of-my-file-when-i-use-semantic-engine| @@ -182,8 +188,9 @@ local binary folder (for example '/usr/local/bin/mvim') and then symlink it: Install YouCompleteMe with Vundle [11]. **Remember:** YCM is a plugin with a compiled component. If you **update** YCM -using Vundle and the ycm_core library API has changed (happens rarely), YCM -will notify you to recompile it. You should then rerun the install process. +using Vundle and the ycm_support_libs library APIs have changed (happens +rarely), YCM will notify you to recompile it. You should then rerun the install +process. It's recommended that you have the latest Xcode installed along with the latest Command Line Tools (that you install from within Xcode). @@ -230,8 +237,9 @@ from source [14] (don't worry, it's easy). Install YouCompleteMe with Vundle [11]. **Remember:** YCM is a plugin with a compiled component. If you **update** YCM -using Vundle and the ycm_core library API has changed (happens rarely), YCM -will notify you to recompile it. You should then rerun the install process. +using Vundle and the ycm_support_libs library APIs have changed (happens +rarely), YCM will notify you to recompile it. You should then rerun the install +process. Install development tools and CMake: 'sudo apt-get install build-essential cmake' @@ -281,8 +289,9 @@ that platform). See the _FAQ_ if you have any issues. **Remember:** YCM is a plugin with a compiled component. If you **update** YCM -using Vundle and the ycm_core library API has changed (happens rarely), YCM -will notify you to recompile it. You should then rerun the install process. +using Vundle and the ycm_support_libs library APIs have changed (happens +rarely), YCM will notify you to recompile it. You should then rerun the install +process. **Please follow the instructions carefully. Read EVERY WORD.** @@ -319,8 +328,8 @@ will notify you to recompile it. You should then rerun the install process. official binaries from llvm.org [18] if at all possible. Make sure you download the correct archive file for your OS. -4. **Compile the 'ycm_core' plugin plugin** (ha!) that YCM needs. This is - the C++ engine that YCM uses to get fast completions. +4. **Compile the 'ycm_support_libs' libraries** that YCM needs. These libs + are the C++ engines that YCM uses to get fast completions. You will need to have 'cmake' installed in order to generate the required makefiles. Linux users can install cmake with their package manager @@ -359,7 +368,7 @@ will notify you to recompile it. You should then rerun the install process. < Now that makefiles have been generated, simply run: > - make ycm_core + make ycm_support_libs < For those who want to use the system version of libclang, you would pass '-DUSE_SYSTEM_LIBCLANG=ON' to cmake _instead of_ the @@ -427,6 +436,15 @@ YCM automatically detects which completion engine would be the best in any situation. On occasion, it queries several of them at once, merges the outputs and presents the results to you. +------------------------------------------------------------------------------- + *youcompleteme-client-server-architecture* +Client-server architecture ~ + +YCM has a client-server architecture; the Vim part of YCM is only a thin client +that talks to the 'ycmd' HTTP+JSON server that has the vast majority of YCM +logic and functionality. The server is started and stopped automatically as you +start and stop Vim. + ------------------------------------------------------------------------------- *youcompleteme-completion-string-ranking* Completion string ranking ~ @@ -624,6 +642,12 @@ yours truly. *youcompleteme-commands* Commands ~ +------------------------------------------------------------------------------- +The *:YcmRestartServer* command + +If the 'ycmd' completion server suddenly stops for some reason, you can restart +it with this command. + ------------------------------------------------------------------------------- The *:YcmForceCompileAndDiagnostics* command @@ -694,7 +718,7 @@ The *GoToDeclaration* subcommand Looks up the symbol under the cursor and jumps to its declaration. -Supported in filetypes: 'c, cpp, objc, objcpp, python' +Supported in filetypes: 'c, cpp, objc, objcpp, python, cs' ------------------------------------------------------------------------------- The *GoToDefinition* subcommand @@ -706,7 +730,7 @@ when the definition of the symbol is in the current translation unit. A translation unit consists of the file you are editing and all the files you are including with '#include' directives (directly or indirectly) in that file. -Supported in filetypes: 'c, cpp, objc, objcpp, python' +Supported in filetypes: 'c, cpp, objc, objcpp, python, cs' ------------------------------------------------------------------------------- The *GoToDefinitionElseDeclaration* subcommand @@ -715,7 +739,7 @@ Looks up the symbol under the cursor and jumps to its definition if possible; if the definition is not accessible from the current translation unit, jumps to the symbol's declaration. -Supported in filetypes: 'c, cpp, objc, objcpp, python' +Supported in filetypes: 'c, cpp, objc, objcpp, python, cs' ------------------------------------------------------------------------------- The *ClearCompilationFlagCache* subcommand @@ -982,6 +1006,64 @@ Default: '0' let g:ycm_seed_identifiers_with_syntax = 0 < ------------------------------------------------------------------------------- +The *g:ycm_server_use_vim_stdout* option + +By default, the 'ycmd' completion server writes logs to logfiles. When this +option is set to '1', the server writes logs to Vim's stdout (so you'll see +them in the console). + +Default: '0' +> + let g:ycm_server_use_vim_stdout = 0 +< +------------------------------------------------------------------------------- +The *g:ycm_server_keep_logfiles* option + +When this option is set to '1', the 'ycmd' completion server will keep the +logfiles around after shutting down (they are deleted on shutdown by default). + +To see where the logfiles are, call |:YcmDebugInfo|. + +Default: '0' +> + let g:ycm_server_keep_logfiles = 0 +< +------------------------------------------------------------------------------- +The *g:ycm_server_log_level* option + +The logging level that the 'ycmd' completion server uses. Valid values are the +following, from most verbose to least verbose: - 'debug' - 'info' - 'warning' - +'error' - 'critical' + +Note that 'debug' is _very_ verbose. + +Default: 'info' +> + let g:ycm_server_log_level = 'info' +< +------------------------------------------------------------------------------- +The *g:ycm_server_idle_suicide_seconds* option + +This option sets the number of seconds of 'ycmd' server idleness (no requests +received) after which the server stops itself. NOTE: the YCM Vim client sends a +shutdown request to the server when Vim is shutting down. + +If your Vim crashes for instance, 'ycmd' never gets the shutdown command and +becomes a zombie process. This option prevents such zombies from sticking +around forever. + +The default option is '43200' seconds which is 12 hours. The reason for the +interval being this long is to prevent the server from shutting down if you +leave your computer (and Vim) turned on during the night. + +The server "heartbeat" that checks whether this interval has passed occurs +every 10 minutes. + +Default: '43200' +> + let g:ycm_server_idle_suicide_seconds = 43200 +< +------------------------------------------------------------------------------- The *g:ycm_csharp_server_port* option The port number (on 'localhost') on which the OmniSharp server should be @@ -1575,6 +1657,8 @@ License ~ This software is licensed under the GPL v3 license [31]. © 2012 Strahinja Val Markovic . + Image: Bitdeli Badge [32] + =============================================================================== *youcompleteme-references* References ~ @@ -1610,5 +1694,7 @@ References ~ [29] https://groups.google.com/forum/?hl=en#!forum/ycm-users [30] https://github.com/Valloric/YouCompleteMe/issues?state=open [31] http://www.gnu.org/copyleft/gpl.html +[32] https://bitdeli.com/free +[33] https://d2weczhvl823v0.cloudfront.net/Valloric/youcompleteme/trend.png vim: ft=help diff --git a/install.sh b/install.sh index 273a8478..c771e4a5 100755 --- a/install.sh +++ b/install.sh @@ -77,7 +77,7 @@ function install { cmake -G "Unix Makefiles" "$@" . $ycm_dir/cpp fi - make -j $(num_cores) ycm_core + make -j $(num_cores) ycm_support_libs popd rm -rf $build_dir } @@ -106,6 +106,22 @@ function usage { exit 0 } +function check_third_party_libs { + libs_present=true + for folder in third_party/*; do + num_files_in_folder=$(find $folder -maxdepth 1 -mindepth 1 | wc -l) + if [[ $num_files_in_folder -eq 0 ]]; then + libs_present=false + fi + done + + if ! $libs_present; then + echo "Some folders in ./third_party are empty; you probably forgot to run:" + printf "\n\tgit submodule update --init --recursive\n\n" + exit 1 + fi +} + cmake_args="" omnisharp_completer=false for flag in $@; do @@ -130,6 +146,8 @@ if [[ $cmake_args == *-DUSE_SYSTEM_LIBCLANG=ON* ]] && \ usage fi +check_third_party_libs + if ! command_exists cmake; then echo "CMake is required to build YouCompleteMe." cmake_install @@ -138,7 +156,7 @@ fi if [ -z "$YCM_TESTRUN" ]; then install $cmake_args $EXTRA_CMAKE_ARGS else - testrun $cmake_args -DUSE_DEV_FLAGS=ON $EXTRA_CMAKE_ARGS + testrun $cmake_args $EXTRA_CMAKE_ARGS fi if $omnisharp_completer; then diff --git a/plugin/youcompleteme.vim b/plugin/youcompleteme.vim index d3c2ee3e..f8e9b3e9 100644 --- a/plugin/youcompleteme.vim +++ b/plugin/youcompleteme.vim @@ -37,11 +37,14 @@ let s:script_folder_path = escape( expand( ':p:h' ), '\' ) function! s:HasYcmCore() let path_prefix = s:script_folder_path . '/../python/' - if filereadable(path_prefix . 'ycm_core.so') + if filereadable(path_prefix . 'ycm_client_support.so') && + \ filereadable(path_prefix . 'ycm_core.so') return 1 - elseif filereadable(path_prefix . 'ycm_core.pyd') + elseif filereadable(path_prefix . 'ycm_client_support.pyd') && + \ filereadable(path_prefix . 'ycm_core.pyd') return 1 - elseif filereadable(path_prefix . 'ycm_core.dll') + elseif filereadable(path_prefix . 'ycm_client_support.dll') && + \ filereadable(path_prefix . 'ycm_core.dll') return 1 endif return 0 @@ -52,7 +55,8 @@ let g:ycm_check_if_ycm_core_present = if g:ycm_check_if_ycm_core_present && !s:HasYcmCore() echohl WarningMsg | - \ echomsg "ycm_core.[so|pyd|dll] not detected; you need to compile " . + \ echomsg "ycm_client_support.[so|pyd|dll] and " . + \ "ycm_core.[so|pyd|dll] not detected; you need to compile " . \ "YCM before using it. Read the docs!" | \ echohl None finish @@ -60,31 +64,11 @@ endif let g:loaded_youcompleteme = 1 -let g:ycm_min_num_of_chars_for_completion = - \ get( g:, 'ycm_min_num_of_chars_for_completion', 2 ) - -let g:ycm_min_num_identifier_candidate_chars = - \ get( g:, 'ycm_min_num_identifier_candidate_chars', 0 ) - -let g:ycm_filetype_whitelist = - \ get( g:, 'ycm_filetype_whitelist', { - \ '*' : 1, - \ } ) - -" The fallback to g:ycm_filetypes_to_completely_ignore is here because of -" backwards compatibility with previous versions of YCM. -let g:ycm_filetype_blacklist = - \ get( g:, 'ycm_filetype_blacklist', - \ get( g:, 'ycm_filetypes_to_completely_ignore', { - \ 'notes' : 1, - \ 'markdown' : 1, - \ 'text' : 1, - \ 'unite' : 1, - \ 'tagbar' : 1, - \ } ) ) - -let g:ycm_filetype_specific_completion_to_disable = - \ get( g:, 'ycm_filetype_specific_completion_to_disable', {} ) +" NOTE: Most defaults are in default_settings.json. They are loaded into Vim +" global with the 'ycm_' prefix if such a key does not already exist; thus, the +" user can override the defaults. +" The only defaults that are here are the ones that are only relevant to the YCM +" Vim client and not the server. let g:ycm_register_as_syntastic_checker = \ get( g:, 'ycm_register_as_syntastic_checker', 1 ) @@ -95,30 +79,12 @@ let g:ycm_allow_changing_updatetime = let g:ycm_add_preview_to_completeopt = \ get( g:, 'ycm_add_preview_to_completeopt', 0 ) -let g:ycm_complete_in_comments = - \ get( g:, 'ycm_complete_in_comments', 0 ) - -let g:ycm_complete_in_strings = - \ get( g:, 'ycm_complete_in_strings', 1 ) - -let g:ycm_collect_identifiers_from_comments_and_strings = - \ get( g:, 'ycm_collect_identifiers_from_comments_and_strings', 0 ) - -let g:ycm_collect_identifiers_from_tags_files = - \ get( g:, 'ycm_collect_identifiers_from_tags_files', 0 ) - -let g:ycm_seed_identifiers_with_syntax = - \ get( g:, 'ycm_seed_identifiers_with_syntax', 0 ) - let g:ycm_autoclose_preview_window_after_completion = \ get( g:, 'ycm_autoclose_preview_window_after_completion', 0 ) let g:ycm_autoclose_preview_window_after_insertion = \ get( g:, 'ycm_autoclose_preview_window_after_insertion', 0 ) -let g:ycm_max_diagnostics_to_display = - \ get( g:, 'ycm_max_diagnostics_to_display', 30 ) - let g:ycm_key_list_select_completion = \ get( g:, 'ycm_key_list_select_completion', ['', ''] ) @@ -131,34 +97,21 @@ let g:ycm_key_invoke_completion = let g:ycm_key_detailed_diagnostics = \ get( g:, 'ycm_key_detailed_diagnostics', 'd' ) -let g:ycm_global_ycm_extra_conf = - \ get( g:, 'ycm_global_ycm_extra_conf', '' ) - -let g:ycm_confirm_extra_conf = - \ get( g:, 'ycm_confirm_extra_conf', 1 ) - -let g:ycm_extra_conf_globlist = - \ get( g:, 'ycm_extra_conf_globlist', [] ) - -let g:ycm_filepath_completion_use_working_dir = - \ get( g:, 'ycm_filepath_completion_use_working_dir', 0 ) - -" Default semantic triggers are in python/ycm/completers/completer.py, these -" just append new triggers to the default dict. -let g:ycm_semantic_triggers = - \ get( g:, 'ycm_semantic_triggers', {} ) - let g:ycm_cache_omnifunc = \ get( g:, 'ycm_cache_omnifunc', 1 ) -let g:ycm_auto_start_csharp_server = - \ get( g:, 'ycm_auto_start_csharp_server', 1 ) +let g:ycm_server_use_vim_stdout = + \ get( g:, 'ycm_server_use_vim_stdout', 0 ) -let g:ycm_auto_stop_csharp_server = - \ get( g:, 'ycm_auto_stop_csharp_server', 1 ) +let g:ycm_server_log_level = + \ get( g:, 'ycm_server_log_level', 'info' ) + +let g:ycm_server_keep_logfiles = + \ get( g:, 'ycm_server_keep_logfiles', 0 ) + +let g:ycm_server_idle_suicide_seconds = + \ get( g:, 'ycm_server_idle_suicide_seconds', 43200 ) -let g:ycm_csharp_server_port = - \ get( g:, 'ycm_csharp_server_port', 2000 ) " On-demand loading. Let's use the autoload folder and not slow down vim's " startup procedure. diff --git a/print_todos.sh b/print_todos.sh new file mode 100755 index 00000000..66e0a456 --- /dev/null +++ b/print_todos.sh @@ -0,0 +1,8 @@ +#!/bin/bash +ag \ +--ignore gmock \ +--ignore jedi/ \ +--ignore OmniSharpServer \ +--ignore testdata \ +TODO \ +cpp/ycm python autoload plugin diff --git a/python/test_requirements.txt b/python/test_requirements.txt index 65164842..1170bb4d 100644 --- a/python/test_requirements.txt +++ b/python/test_requirements.txt @@ -1,3 +1,6 @@ flake8>=2.0 mock>=1.0.1 nose>=1.3.0 +PyHamcrest>=1.7.2 +WebTest>=2.0.9 + diff --git a/python/ycm/base.py b/python/ycm/base.py index 8082b0e8..e578fa3b 100644 --- a/python/ycm/base.py +++ b/python/ycm/base.py @@ -17,21 +17,42 @@ # You should have received a copy of the GNU General Public License # along with YouCompleteMe. If not, see . -import os import re import vim from ycm import vimsupport from ycm import utils +from ycm import user_options_store +import ycm_client_support -try: - import ycm_core -except ImportError as e: - vimsupport.PostVimMessage( - 'Error importing ycm_core. Are you sure you have placed a version 3.2+ ' - 'libclang.[so|dll|dylib] in folder "{0}"? See the Installation Guide in ' - 'the docs. Full error: {1}'.format( - os.path.dirname( os.path.dirname( os.path.abspath( __file__ ) ) ), - str( e ) ) ) +YCM_VAR_PREFIX = 'ycm_' + + +def BuildServerConf(): + """Builds a dictionary mapping YCM Vim user options to values. Option names + don't have the 'ycm_' prefix.""" + + vim_globals = vimsupport.GetReadOnlyVimGlobals( force_python_objects = True ) + server_conf = {} + for key, value in vim_globals.items(): + if not key.startswith( YCM_VAR_PREFIX ): + continue + try: + new_value = int( value ) + except: + new_value = value + new_key = key[ len( YCM_VAR_PREFIX ): ] + server_conf[ new_key ] = new_value + + return server_conf + + +def LoadJsonDefaultsIntoVim(): + defaults = user_options_store.DefaultOptions() + vim_defaults = {} + for key, value in defaults.iteritems(): + vim_defaults[ 'ycm_' + key ] = value + + vimsupport.LoadDictIntoVimGlobals( vim_defaults, overwrite = False ) def CompletionStartColumn(): @@ -121,11 +142,11 @@ def AdjustCandidateInsertionText( candidates ): return new_candidates -COMPATIBLE_WITH_CORE_VERSION = 4 +COMPATIBLE_WITH_CORE_VERSION = 7 def CompatibleWithYcmCore(): try: - current_core_version = ycm_core.YcmCoreVersion() + current_core_version = ycm_client_support.YcmCoreVersion() except AttributeError: return False diff --git a/python/ycm/client/__init__.py b/python/ycm/client/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/python/ycm/client/base_request.py b/python/ycm/client/base_request.py new file mode 100644 index 00000000..ad3f1fbf --- /dev/null +++ b/python/ycm/client/base_request.py @@ -0,0 +1,154 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +import vim +import json +import requests +import urlparse +from retries import retries +from requests_futures.sessions import FuturesSession +from concurrent.futures import ThreadPoolExecutor +from ycm import vimsupport +from ycm.server.responses import ServerError, UnknownExtraConf + +HEADERS = {'content-type': 'application/json'} +EXECUTOR = ThreadPoolExecutor( max_workers = 10 ) +# Setting this to None seems to screw up the Requests/urllib3 libs. +DEFAULT_TIMEOUT_SEC = 30 + +class BaseRequest( object ): + def __init__( self ): + pass + + + def Start( self ): + pass + + + def Done( self ): + return True + + + def Response( self ): + return {} + + + # This is the blocking version of the method. See below for async. + # |timeout| is num seconds to tolerate no response from server before giving + # up; see Requests docs for details (we just pass the param along). + @staticmethod + def PostDataToHandler( data, handler, timeout = DEFAULT_TIMEOUT_SEC ): + return JsonFromFuture( BaseRequest.PostDataToHandlerAsync( data, + handler, + timeout ) ) + + + # This returns a future! Use JsonFromFuture to get the value. + # |timeout| is num seconds to tolerate no response from server before giving + # up; see Requests docs for details (we just pass the param along). + @staticmethod + def PostDataToHandlerAsync( data, handler, timeout = DEFAULT_TIMEOUT_SEC ): + def PostData( data, handler, timeout ): + return BaseRequest.session.post( _BuildUri( handler ), + data = json.dumps( data ), + headers = HEADERS, + timeout = timeout ) + + @retries( 5, delay = 0.5, backoff = 1.5 ) + def DelayedPostData( data, handler ): + return requests.post( _BuildUri( handler ), + data = json.dumps( data ), + headers = HEADERS ) + + if not _CheckServerIsHealthyWithCache(): + return EXECUTOR.submit( DelayedPostData, data, handler ) + + return PostData( data, handler, timeout ) + + + session = FuturesSession( executor = EXECUTOR ) + server_location = 'http://localhost:6666' + + +def BuildRequestData( start_column = None, + query = None, + include_buffer_data = True ): + line, column = vimsupport.CurrentLineAndColumn() + filepath = vimsupport.GetCurrentBufferFilepath() + request_data = { + 'filetypes': vimsupport.CurrentFiletypes(), + 'line_num': line, + 'column_num': column, + 'start_column': start_column, + 'line_value': vim.current.line, + 'filepath': filepath + } + + if include_buffer_data: + request_data[ 'file_data' ] = vimsupport.GetUnsavedAndCurrentBufferData() + if query: + request_data[ 'query' ] = query + + return request_data + + +def JsonFromFuture( future ): + response = future.result() + if response.status_code == requests.codes.server_error: + _RaiseExceptionForData( response.json() ) + + # We let Requests handle the other status types, we only handle the 500 + # error code. + response.raise_for_status() + + if response.text: + return response.json() + return None + + +def _BuildUri( handler ): + return urlparse.urljoin( BaseRequest.server_location, handler ) + + +SERVER_HEALTHY = False + +def _CheckServerIsHealthyWithCache(): + global SERVER_HEALTHY + + def _ServerIsHealthy(): + response = requests.get( _BuildUri( 'healthy' ) ) + response.raise_for_status() + return response.json() + + if SERVER_HEALTHY: + return True + + try: + SERVER_HEALTHY = _ServerIsHealthy() + return SERVER_HEALTHY + except: + return False + + +def _RaiseExceptionForData( data ): + if data[ 'exception' ][ 'TYPE' ] == UnknownExtraConf.__name__: + raise UnknownExtraConf( data[ 'exception' ][ 'extra_conf_file' ] ) + + raise ServerError( '{0}: {1}'.format( data[ 'exception' ][ 'TYPE' ], + data[ 'message' ] ) ) diff --git a/python/ycm/client/command_request.py b/python/ycm/client/command_request.py new file mode 100644 index 00000000..f2b92ab3 --- /dev/null +++ b/python/ycm/client/command_request.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +import vim +from ycm.client.base_request import BaseRequest, BuildRequestData, ServerError +from ycm import vimsupport +from ycm.utils import ToUtf8IfNeeded + + +class CommandRequest( BaseRequest ): + def __init__( self, arguments, completer_target = None ): + super( CommandRequest, self ).__init__() + self._arguments = arguments + self._completer_target = ( completer_target if completer_target + else 'filetype_default' ) + self._is_goto_command = ( + arguments and arguments[ 0 ].startswith( 'GoTo' ) ) + self._response = None + + + def Start( self ): + request_data = BuildRequestData() + request_data.update( { + 'completer_target': self._completer_target, + 'command_arguments': self._arguments + } ) + try: + self._response = self.PostDataToHandler( request_data, + 'run_completer_command' ) + except ServerError as e: + vimsupport.PostVimMessage( e ) + + + def Response( self ): + return self._response + + + def RunPostCommandActionsIfNeeded( self ): + if not self._is_goto_command or not self.Done() or not self._response: + return + + if isinstance( self._response, list ): + defs = [ _BuildQfListItem( x ) for x in self._response ] + vim.eval( 'setqflist( %s )' % repr( defs ) ) + vim.eval( 'youcompleteme#OpenGoToList()' ) + else: + vimsupport.JumpToLocation( self._response[ 'filepath' ], + self._response[ 'line_num' ] + 1, + self._response[ 'column_num' ] + 1) + + + + +def SendCommandRequest( arguments, completer ): + request = CommandRequest( arguments, completer ) + # This is a blocking call. + request.Start() + request.RunPostCommandActionsIfNeeded() + return request.Response() + + +def _BuildQfListItem( goto_data_item ): + qf_item = {} + if 'filepath' in goto_data_item: + qf_item[ 'filename' ] = ToUtf8IfNeeded( goto_data_item[ 'filepath' ] ) + if 'description' in goto_data_item: + qf_item[ 'text' ] = ToUtf8IfNeeded( goto_data_item[ 'description' ] ) + if 'line_num' in goto_data_item: + qf_item[ 'lnum' ] = goto_data_item[ 'line_num' ] + 1 + if 'column_num' in goto_data_item: + qf_item[ 'col' ] = goto_data_item[ 'column_num' ] + return qf_item diff --git a/python/ycm/client/completion_request.py b/python/ycm/client/completion_request.py new file mode 100644 index 00000000..bb0001c2 --- /dev/null +++ b/python/ycm/client/completion_request.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +from ycm import base +from ycm import vimsupport +from ycm.client.base_request import ( BaseRequest, BuildRequestData, + JsonFromFuture ) + +TIMEOUT_SECONDS = 0.5 + +class CompletionRequest( BaseRequest ): + def __init__( self, force_semantic = False ): + super( CompletionRequest, self ).__init__() + + self._completion_start_column = base.CompletionStartColumn() + + # This field is also used by the omni_completion_request subclass + self.request_data = BuildRequestData( self._completion_start_column ) + if force_semantic: + self.request_data[ 'force_semantic' ] = True + + + def CompletionStartColumn( self ): + return self._completion_start_column + + + def Start( self, query ): + self.request_data[ 'query' ] = query + self._response_future = self.PostDataToHandlerAsync( self.request_data, + 'completions', + TIMEOUT_SECONDS ) + + + def Done( self ): + return self._response_future.done() + + + def Response( self ): + if not self._response_future: + return [] + try: + return [ _ConvertCompletionDataToVimData( x ) + for x in JsonFromFuture( self._response_future ) ] + except Exception as e: + vimsupport.PostVimMessage( str( e ) ) + return [] + + +def _ConvertCompletionDataToVimData( completion_data ): + # see :h complete-items for a description of the dictionary fields + vim_data = { + 'word' : completion_data[ 'insertion_text' ], + 'dup' : 1, + } + + if 'menu_text' in completion_data: + vim_data[ 'abbr' ] = completion_data[ 'menu_text' ] + if 'extra_menu_info' in completion_data: + vim_data[ 'menu' ] = completion_data[ 'extra_menu_info' ] + if 'kind' in completion_data: + vim_data[ 'kind' ] = completion_data[ 'kind' ] + if 'detailed_info' in completion_data: + vim_data[ 'info' ] = completion_data[ 'detailed_info' ] + + return vim_data diff --git a/python/ycm/client/event_notification.py b/python/ycm/client/event_notification.py new file mode 100644 index 00000000..9652413f --- /dev/null +++ b/python/ycm/client/event_notification.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +from ycm import vimsupport +from ycm.server.responses import UnknownExtraConf +from ycm.client.base_request import ( BaseRequest, BuildRequestData, + JsonFromFuture ) + + +class EventNotification( BaseRequest ): + def __init__( self, event_name, extra_data = None ): + super( EventNotification, self ).__init__() + self._event_name = event_name + self._extra_data = extra_data + self._cached_response = None + + + def Start( self ): + request_data = BuildRequestData() + if self._extra_data: + request_data.update( self._extra_data ) + request_data[ 'event_name' ] = self._event_name + + self._response_future = self.PostDataToHandlerAsync( request_data, + 'event_notification' ) + + + def Done( self ): + return self._response_future.done() + + + def Response( self ): + if self._cached_response: + return self._cached_response + + if not self._response_future or self._event_name != 'FileReadyToParse': + return [] + + try: + try: + self._cached_response = JsonFromFuture( self._response_future ) + except UnknownExtraConf as e: + if vimsupport.Confirm( str( e ) ): + _LoadExtraConfFile( e.extra_conf_file ) + except Exception as e: + vimsupport.PostVimMessage( str( e ) ) + + if not self._cached_response: + return [] + + self._cached_response = [ _ConvertDiagnosticDataToVimData( x ) + for x in self._cached_response ] + return self._cached_response + + +def _ConvertDiagnosticDataToVimData( diagnostic ): + # see :h getqflist for a description of the dictionary fields + # Note that, as usual, Vim is completely inconsistent about whether + # line/column numbers are 1 or 0 based in its various APIs. Here, it wants + # them to be 1-based. + return { + 'bufnr' : vimsupport.GetBufferNumberForFilename( diagnostic[ 'filepath' ]), + 'lnum' : diagnostic[ 'line_num' ] + 1, + 'col' : diagnostic[ 'column_num' ] + 1, + 'text' : diagnostic[ 'text' ], + 'type' : diagnostic[ 'kind' ], + 'valid' : 1 + } + + +def SendEventNotificationAsync( event_name, extra_data = None ): + event = EventNotification( event_name, extra_data ) + event.Start() + +def _LoadExtraConfFile( filepath ): + BaseRequest.PostDataToHandler( { 'filepath': filepath }, + 'load_extra_conf_file' ) diff --git a/python/ycm/client/omni_completion_request.py b/python/ycm/client/omni_completion_request.py new file mode 100644 index 00000000..2eb26a41 --- /dev/null +++ b/python/ycm/client/omni_completion_request.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +from ycm.client.completion_request import CompletionRequest + + +class OmniCompletionRequest( CompletionRequest ): + def __init__( self, omni_completer ): + super( OmniCompletionRequest, self ).__init__() + self._omni_completer = omni_completer + + + def Start( self, query ): + self.request_data[ 'query' ] = query + self._results = self._omni_completer.ComputeCandidates( self.request_data ) + + + def Done( self ): + return True + + + def Response( self ): + return self._results diff --git a/python/ycm/completers/all/identifier_completer.py b/python/ycm/completers/all/identifier_completer.py index 932b4406..c3e27bfb 100644 --- a/python/ycm/completers/all/identifier_completer.py +++ b/python/ycm/completers/all/identifier_completer.py @@ -18,175 +18,154 @@ # along with YouCompleteMe. If not, see . import os -import vim +import logging import ycm_core from collections import defaultdict from ycm.completers.general_completer import GeneralCompleter -from ycm.completers.general import syntax_parse -from ycm import vimsupport +# from ycm.completers.general import syntax_parse from ycm import utils +from ycm.utils import ToUtf8IfNeeded +from ycm.server import responses MAX_IDENTIFIER_COMPLETIONS_RETURNED = 10 -MIN_NUM_COMPLETION_START_CHARS = int( vimsupport.GetVariableValue( - "g:ycm_min_num_of_chars_for_completion" ) ) -MIN_NUM_CANDIDATE_SIZE_CHARS = int( vimsupport.GetVariableValue( - "g:ycm_min_num_identifier_candidate_chars" ) ) SYNTAX_FILENAME = 'YCM_PLACEHOLDER_FOR_SYNTAX' class IdentifierCompleter( GeneralCompleter ): - def __init__( self ): - super( IdentifierCompleter, self ).__init__() - self.completer = ycm_core.IdentifierCompleter() - self.completer.EnableThreading() - self.tags_file_last_mtime = defaultdict( int ) - self.filetypes_with_keywords_loaded = set() + def __init__( self, user_options ): + super( IdentifierCompleter, self ).__init__( user_options ) + self._completer = ycm_core.IdentifierCompleter() + self._tags_file_last_mtime = defaultdict( int ) + self._logger = logging.getLogger( __name__ ) - def ShouldUseNow( self, start_column ): - return self.QueryLengthAboveMinThreshold( start_column ) + def ShouldUseNow( self, request_data ): + return self.QueryLengthAboveMinThreshold( request_data ) - def CandidatesForQueryAsync( self, query, unused_start_column ): - filetype = vim.eval( "&filetype" ) - self.completions_future = self.completer.CandidatesForQueryAndTypeAsync( - utils.SanitizeQuery( query ), - filetype ) + def ComputeCandidates( self, request_data ): + if not self.ShouldUseNow( request_data ): + return [] + + completions = self._completer.CandidatesForQueryAndType( + ToUtf8IfNeeded( utils.SanitizeQuery( request_data[ 'query' ] ) ), + ToUtf8IfNeeded( request_data[ 'filetypes' ][ 0 ] ) ) + + completions = completions[ : MAX_IDENTIFIER_COMPLETIONS_RETURNED ] + completions = _RemoveSmallCandidates( + completions, self.user_options[ 'min_num_identifier_candidate_chars' ] ) + + return [ responses.BuildCompletionData( x ) for x in completions ] - def AddIdentifier( self, identifier ): - filetype = vim.eval( "&filetype" ) - filepath = vim.eval( "expand('%:p')" ) + def AddIdentifier( self, identifier, request_data ): + filetype = request_data[ 'filetypes' ][ 0 ] + filepath = request_data[ 'filepath' ] if not filetype or not filepath or not identifier: return vector = ycm_core.StringVec() - vector.append( identifier ) - self.completer.AddIdentifiersToDatabase( vector, - filetype, - filepath ) + vector.append( ToUtf8IfNeeded( identifier ) ) + self._logger.info( 'Adding ONE buffer identifier for file: %s', filepath ) + self._completer.AddIdentifiersToDatabase( vector, + ToUtf8IfNeeded( filetype ), + ToUtf8IfNeeded( filepath ) ) - def AddPreviousIdentifier( self ): - self.AddIdentifier( PreviousIdentifier() ) + def AddPreviousIdentifier( self, request_data ): + self.AddIdentifier( + _PreviousIdentifier( + self.user_options[ 'min_num_of_chars_for_completion' ], + request_data ), + request_data ) - def AddIdentifierUnderCursor( self ): - cursor_identifier = vim.eval( 'expand("")' ) + def AddIdentifierUnderCursor( self, request_data ): + cursor_identifier = _GetCursorIdentifier( request_data ) if not cursor_identifier: return - stripped_cursor_identifier = ''.join( ( x for x in - cursor_identifier if - utils.IsIdentifierChar( x ) ) ) - if not stripped_cursor_identifier: - return - - self.AddIdentifier( stripped_cursor_identifier ) + self.AddIdentifier( cursor_identifier, request_data ) - def AddBufferIdentifiers( self ): - # TODO: use vimsupport.GetFiletypes; also elsewhere in file - filetype = vim.eval( "&filetype" ) - filepath = vim.eval( "expand('%:p')" ) - collect_from_comments_and_strings = vimsupport.GetBoolValue( - "g:ycm_collect_identifiers_from_comments_and_strings" ) + def AddBufferIdentifiers( self, request_data ): + filetype = request_data[ 'filetypes' ][ 0 ] + filepath = request_data[ 'filepath' ] + collect_from_comments_and_strings = bool( self.user_options[ + 'collect_identifiers_from_comments_and_strings' ] ) if not filetype or not filepath: return - text = "\n".join( vim.current.buffer ) - self.completer.AddIdentifiersToDatabaseFromBufferAsync( - text, - filetype, - filepath, + text = request_data[ 'file_data' ][ filepath ][ 'contents' ] + self._logger.info( 'Adding buffer identifiers for file: %s', filepath ) + self._completer.AddIdentifiersToDatabaseFromBuffer( + ToUtf8IfNeeded( text ), + ToUtf8IfNeeded( filetype ), + ToUtf8IfNeeded( filepath ), collect_from_comments_and_strings ) - def AddIdentifiersFromTagFiles( self ): - tag_files = vim.eval( 'tagfiles()' ) - current_working_directory = os.getcwd() + def AddIdentifiersFromTagFiles( self, tag_files ): absolute_paths_to_tag_files = ycm_core.StringVec() for tag_file in tag_files: - absolute_tag_file = os.path.join( current_working_directory, - tag_file ) try: - current_mtime = os.path.getmtime( absolute_tag_file ) + current_mtime = os.path.getmtime( tag_file ) except: continue - last_mtime = self.tags_file_last_mtime[ absolute_tag_file ] + last_mtime = self._tags_file_last_mtime[ tag_file ] # We don't want to repeatedly process the same file over and over; we only # process if it's changed since the last time we looked at it if current_mtime <= last_mtime: continue - self.tags_file_last_mtime[ absolute_tag_file ] = current_mtime - absolute_paths_to_tag_files.append( absolute_tag_file ) + self._tags_file_last_mtime[ tag_file ] = current_mtime + absolute_paths_to_tag_files.append( ToUtf8IfNeeded( tag_file ) ) if not absolute_paths_to_tag_files: return - self.completer.AddIdentifiersToDatabaseFromTagFilesAsync( + self._completer.AddIdentifiersToDatabaseFromTagFiles( absolute_paths_to_tag_files ) - def AddIdentifiersFromSyntax( self ): - filetype = vim.eval( "&filetype" ) - if filetype in self.filetypes_with_keywords_loaded: - return + def AddIdentifiersFromSyntax( self, keyword_list, filetypes ): + keyword_vector = ycm_core.StringVec() + for keyword in keyword_list: + keyword_vector.append( ToUtf8IfNeeded( keyword ) ) - self.filetypes_with_keywords_loaded.add( filetype ) - - keyword_set = syntax_parse.SyntaxKeywordsForCurrentBuffer() - keywords = ycm_core.StringVec() - for keyword in keyword_set: - keywords.append( keyword ) - - filepath = SYNTAX_FILENAME + filetype - self.completer.AddIdentifiersToDatabase( keywords, - filetype, - filepath ) + filepath = SYNTAX_FILENAME + filetypes[ 0 ] + self._completer.AddIdentifiersToDatabase( keyword_vector, + ToUtf8IfNeeded( filetypes[ 0 ] ), + ToUtf8IfNeeded( filepath ) ) - def OnFileReadyToParse( self ): - self.AddBufferIdentifiers() - - if vimsupport.GetBoolValue( 'g:ycm_collect_identifiers_from_tags_files' ): - self.AddIdentifiersFromTagFiles() - - if vimsupport.GetBoolValue( 'g:ycm_seed_identifiers_with_syntax' ): - self.AddIdentifiersFromSyntax() + def OnFileReadyToParse( self, request_data ): + self.AddBufferIdentifiers( request_data ) + if 'tag_files' in request_data: + self.AddIdentifiersFromTagFiles( request_data[ 'tag_files' ] ) + if 'syntax_keywords' in request_data: + self.AddIdentifiersFromSyntax( request_data[ 'syntax_keywords' ], + request_data[ 'filetypes' ] ) - def OnInsertLeave( self ): - self.AddIdentifierUnderCursor() + def OnInsertLeave( self, request_data ): + self.AddIdentifierUnderCursor( request_data ) - def OnCurrentIdentifierFinished( self ): - self.AddPreviousIdentifier() + def OnCurrentIdentifierFinished( self, request_data ): + self.AddPreviousIdentifier( request_data ) - def CandidatesFromStoredRequest( self ): - if not self.completions_future: - return [] - completions = self.completions_future.GetResults()[ - : MAX_IDENTIFIER_COMPLETIONS_RETURNED ] - - completions = _RemoveSmallCandidates( completions ) - - # We will never have duplicates in completions so with 'dup':1 we tell Vim - # to add this candidate even if it's a duplicate of an existing one (which - # will never happen). This saves us some expensive string matching - # operations in Vim. - return [ { 'word': x, 'dup': 1 } for x in completions ] - - -def PreviousIdentifier(): - line_num, column_num = vimsupport.CurrentLineAndColumn() - buffer = vim.current.buffer - line = buffer[ line_num ] +def _PreviousIdentifier( min_num_completion_start_chars, request_data ): + line_num = request_data[ 'line_num' ] + column_num = request_data[ 'column_num' ] + filepath = request_data[ 'filepath' ] + contents_per_line = ( + request_data[ 'file_data' ][ filepath ][ 'contents' ].split( '\n' ) ) + line = contents_per_line[ line_num ] end_column = column_num @@ -196,7 +175,7 @@ def PreviousIdentifier(): # Look at the previous line if we reached the end of the current one if end_column == 0: try: - line = buffer[ line_num - 1] + line = contents_per_line[ line_num - 1 ] except: return "" end_column = len( line ) @@ -208,15 +187,52 @@ def PreviousIdentifier(): while start_column > 0 and utils.IsIdentifierChar( line[ start_column - 1 ] ): start_column -= 1 - if end_column - start_column < MIN_NUM_COMPLETION_START_CHARS: + if end_column - start_column < min_num_completion_start_chars: return "" return line[ start_column : end_column ] -def _RemoveSmallCandidates( candidates ): - if MIN_NUM_CANDIDATE_SIZE_CHARS == 0: +def _RemoveSmallCandidates( candidates, min_num_candidate_size_chars ): + if min_num_candidate_size_chars == 0: return candidates - return [ x for x in candidates if len( x ) >= MIN_NUM_CANDIDATE_SIZE_CHARS ] + return [ x for x in candidates if len( x ) >= min_num_candidate_size_chars ] + + +# This is meant to behave like 'expand(" 0 and utils.IsIdentifierChar( line[ + identifier_start - 1 ] ): + identifier_start -= 1 + return identifier_start + + + def FindIdentifierEnd( line, valid_char_column ): + identifier_end = valid_char_column + while identifier_end < len( line ) - 1 and utils.IsIdentifierChar( line[ + identifier_end + 1 ] ): + identifier_end += 1 + return identifier_end + 1 + + column_num = request_data[ 'column_num' ] + line = request_data[ 'line_value' ] + + try: + valid_char_column = FindFirstValidChar( line, column_num ) + return line[ FindIdentifierStart( line, valid_char_column ) : + FindIdentifierEnd( line, valid_char_column ) ] + except: + return '' diff --git a/python/ycm/completers/all/omni_completer.py b/python/ycm/completers/all/omni_completer.py index 66b6fff6..5bfffeff 100644 --- a/python/ycm/completers/all/omni_completer.py +++ b/python/ycm/completers/all/omni_completer.py @@ -19,17 +19,18 @@ import vim from ycm import vimsupport +from ycm import base from ycm.completers.completer import Completer +from ycm.client.base_request import BuildRequestData OMNIFUNC_RETURNED_BAD_VALUE = 'Omnifunc returned bad value to YCM!' OMNIFUNC_NOT_LIST = ( 'Omnifunc did not return a list or a dict with a "words" ' ' list when expected.' ) class OmniCompleter( Completer ): - def __init__( self ): - super( OmniCompleter, self ).__init__() - self.omnifunc = None - self.stored_candidates = None + def __init__( self, user_options ): + super( OmniCompleter, self ).__init__( user_options ) + self._omnifunc = None def SupportedFiletypes( self ): @@ -37,76 +38,76 @@ class OmniCompleter( Completer ): def ShouldUseCache( self ): - return vimsupport.GetBoolValue( "g:ycm_cache_omnifunc" ) + return bool( self.user_options[ 'cache_omnifunc' ] ) - def ShouldUseNow( self, start_column ): - if self.ShouldUseCache(): - return super( OmniCompleter, self ).ShouldUseNow( start_column ) - return self.ShouldUseNowInner( start_column ) - - - def ShouldUseNowInner( self, start_column ): - if not self.omnifunc: + # We let the caller call this without passing in request_data. This is useful + # for figuring out should we even be preparing the "real" request_data in + # omni_completion_request. The real request_data is much bigger and takes + # longer to prepare, and we want to avoid creating it twice. + def ShouldUseNow( self, request_data = None ): + if not self._omnifunc: return False - return super( OmniCompleter, self ).ShouldUseNowInner( start_column ) + if not request_data: + request_data = _BuildRequestDataSubstitute() - def CandidatesForQueryAsync( self, query, unused_start_column ): if self.ShouldUseCache(): - return super( OmniCompleter, self ).CandidatesForQueryAsync( - query, unused_start_column ) + return super( OmniCompleter, self ).ShouldUseNow( request_data ) + return self.ShouldUseNowInner( request_data ) + + + def ShouldUseNowInner( self, request_data ): + if not self._omnifunc: + return False + return super( OmniCompleter, self ).ShouldUseNowInner( request_data ) + + + def ComputeCandidates( self, request_data ): + if self.ShouldUseCache(): + return super( OmniCompleter, self ).ComputeCandidates( + request_data ) else: - return self.CandidatesForQueryAsyncInner( query, unused_start_column ) + if self.ShouldUseNowInner( request_data ): + return self.ComputeCandidatesInner( request_data ) + return [] - def CandidatesForQueryAsyncInner( self, query, unused_start_column ): - if not self.omnifunc: - self.stored_candidates = None - return + def ComputeCandidatesInner( self, request_data ): + if not self._omnifunc: + return [] try: - return_value = int( vim.eval( self.omnifunc + '(1,"")' ) ) + return_value = int( vim.eval( self._omnifunc + '(1,"")' ) ) if return_value < 0: - self.stored_candidates = None - return + return [] - omnifunc_call = [ self.omnifunc, + omnifunc_call = [ self._omnifunc, "(0,'", - vimsupport.EscapeForVim( query ), + vimsupport.EscapeForVim( request_data[ 'query' ] ), "')" ] items = vim.eval( ''.join( omnifunc_call ) ) if 'words' in items: - items = items['words'] + items = items[ 'words' ] if not hasattr( items, '__iter__' ): raise TypeError( OMNIFUNC_NOT_LIST ) - self.stored_candidates = filter( bool, items ) - except (TypeError, ValueError) as error: + return filter( bool, items ) + except ( TypeError, ValueError ) as error: vimsupport.PostVimMessage( OMNIFUNC_RETURNED_BAD_VALUE + ' ' + str( error ) ) - self.stored_candidates = None - return + return [] - - def AsyncCandidateRequestReadyInner( self ): - return True + def OnFileReadyToParse( self, request_data ): + self._omnifunc = vim.eval( '&omnifunc' ) - def OnFileReadyToParse( self ): - self.omnifunc = vim.eval( '&omnifunc' ) +def _BuildRequestDataSubstitute(): + data = BuildRequestData( include_buffer_data = False ) + data[ 'start_column' ] = base.CompletionStartColumn() + return data - def CandidatesFromStoredRequest( self ): - if self.ShouldUseCache(): - return super( OmniCompleter, self ).CandidatesFromStoredRequest() - else: - return self.CandidatesFromStoredRequestInner() - - - def CandidatesFromStoredRequestInner( self ): - return self.stored_candidates if self.stored_candidates else [] - diff --git a/python/ycm/completers/all/tests/identifier_completer_test.py b/python/ycm/completers/all/tests/identifier_completer_test.py new file mode 100644 index 00000000..532773cd --- /dev/null +++ b/python/ycm/completers/all/tests/identifier_completer_test.py @@ -0,0 +1,127 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +from nose.tools import eq_ +from ycm.completers.all import identifier_completer + + +def GetCursorIdentifier_StartOfLine_test(): + eq_( 'foo', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 0, + 'line_value': 'foo' + } ) ) + + eq_( 'fooBar', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 0, + 'line_value': 'fooBar' + } ) ) + + +def GetCursorIdentifier_EndOfLine_test(): + eq_( 'foo', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 2, + 'line_value': 'foo' + } ) ) + + +def GetCursorIdentifier_PastEndOfLine_test(): + eq_( '', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 10, + 'line_value': 'foo' + } ) ) + + +def GetCursorIdentifier_NegativeColumn_test(): + eq_( '', + identifier_completer._GetCursorIdentifier( + { + 'column_num': -10, + 'line_value': 'foo' + } ) ) + + +def GetCursorIdentifier_StartOfLine_StopsAtNonIdentifierChar_test(): + eq_( 'foo', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 0, + 'line_value': 'foo(goo)' + } ) ) + + +def GetCursorIdentifier_AtNonIdentifier_test(): + eq_( 'goo', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 3, + 'line_value': 'foo(goo)' + } ) ) + + +def GetCursorIdentifier_WalksForwardForIdentifier_test(): + eq_( 'foo', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 0, + 'line_value': ' foo' + } ) ) + + +def GetCursorIdentifier_FindsNothingForward_test(): + eq_( '', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 4, + 'line_value': 'foo ()***()' + } ) ) + + +def GetCursorIdentifier_SingleCharIdentifier_test(): + eq_( 'f', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 0, + 'line_value': ' f ' + } ) ) + + +def GetCursorIdentifier_StartsInMiddleOfIdentifier_test(): + eq_( 'foobar', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 3, + 'line_value': 'foobar' + } ) ) + + +def GetCursorIdentifier_LineEmpty_test(): + eq_( '', + identifier_completer._GetCursorIdentifier( + { + 'column_num': 11, + 'line_value': '' + } ) ) diff --git a/python/ycm/completers/c/hook.py b/python/ycm/completers/c/hook.py index a4b02e0b..7c6f20e5 100644 --- a/python/ycm/completers/c/hook.py +++ b/python/ycm/completers/c/hook.py @@ -20,9 +20,9 @@ import ycm_core from ycm.completers.cpp.clang_completer import ClangCompleter -def GetCompleter(): +def GetCompleter( user_options ): if ycm_core.HasClangSupport(): - return ClangCompleter() + return ClangCompleter( user_options ) else: return None diff --git a/python/ycm/completers/completer.py b/python/ycm/completers/completer.py index dee07e5f..63ccffa7 100644 --- a/python/ycm/completers/completer.py +++ b/python/ycm/completers/completer.py @@ -18,16 +18,12 @@ # along with YouCompleteMe. If not, see . import abc -import vim -import ycm_core -from ycm import vimsupport +import ycm_client_support +from ycm.utils import ToUtf8IfNeeded, ForceSemanticCompletion from ycm.completers.completer_utils import TriggersForFiletype NO_USER_COMMANDS = 'This completer does not define any commands.' -MIN_NUM_CHARS = int( vimsupport.GetVariableValue( - "g:ycm_min_num_of_chars_for_completion" ) ) - class Completer( object ): """A base class for all Completers in YCM. @@ -36,10 +32,11 @@ class Completer( object ): calling on your Completer: ShouldUseNow() is called with the start column of where a potential completion - string should start. For instance, if the user's input is 'foo.bar' and the - cursor is on the 'r' in 'bar', start_column will be the 0-based index of 'b' - in the line. Your implementation of ShouldUseNow() should return True if your - semantic completer should be used and False otherwise. + string should start and the current line (string) the cursor is on. For + instance, if the user's input is 'foo.bar' and the cursor is on the 'r' in + 'bar', start_column will be the 0-based index of 'b' in the line. Your + implementation of ShouldUseNow() should return True if your semantic completer + should be used and False otherwise. This is important to get right. You want to return False if you can't provide completions because then the identifier completer will kick in, and that's @@ -61,44 +58,21 @@ class Completer( object ): and will NOT re-query your completer but will in fact provide fuzzy-search on the candidate strings that were stored in the cache. - CandidatesForQueryAsync() is the main entry point when the user types. For + ComputeCandidates() is the main entry point when the user types. For "foo.bar", the user query is "bar" and completions matching this string should - be shown. The job of CandidatesForQueryAsync() is to merely initiate this - request, which will hopefully be processed in a background thread. You may - want to subclass ThreadedCompleter instead of Completer directly. + be shown. It should return the list of candidates. The format of the result + can be a list of strings or a more complicated list of dictionaries. Use + ycm.server.responses.BuildCompletionData to build the detailed response. See + clang_completer.py to see how its used in practice. - AsyncCandidateRequestReady() is the function that is repeatedly polled until - it returns True. If CandidatesForQueryAsync() started a background task of - collecting the required completions, AsyncCandidateRequestReady() would check - the state of that task and return False until it was completed. - - CandidatesFromStoredRequest() should return the list of candidates. This is - what YCM calls after AsyncCandidateRequestReady() returns True. The format of - the result can be a list of strings or a more complicated list of - dictionaries. See ':h complete-items' for the format, and clang_completer.py - to see how its used in practice. + Again, you probably want to override ComputeCandidatesInner(). You also need to implement the SupportedFiletypes() function which should return a list of strings, where the strings are Vim filetypes your completer supports. - clang_completer.py is a good example of a "complicated" completer that - maintains its own internal cache and therefore directly overrides the "main" - functions in the API instead of the *Inner versions. A good example of a - simple completer that does not do this is omni_completer.py. - - If you're confident your completer doesn't need a background task (think - again, you probably do) because you can "certainly" furnish a response in - under 10ms, then you can perform your backend processing in a synchronous - fashion. You may also need to do this because of technical restrictions (much - like omni_completer.py has to do it because accessing Vim internals is not - thread-safe). But even if you're certain, still try to do the processing in a - background thread. Your completer is unlikely to be merged if it does not, - because synchronous processing will block Vim's GUI thread and that's a very, - VERY bad thing (so try not to do it!). Again, you may want to subclass - ThreadedCompleter instead of Completer directly; ThreadedCompleter will - abstract away the use of a background thread for you. See - threaded_completer.py. + clang_completer.py is a good example of a "complicated" completer. A good + example of a simple completer is ultisnips_completer.py. The On* functions are provided for your convenience. They are called when their specific events occur. For instance, the identifier completer collects @@ -108,42 +82,48 @@ class Completer( object ): One special function is OnUserCommand. It is called when the user uses the command :YcmCompleter and is passed all extra arguments used on command invocation (e.g. OnUserCommand(['first argument', 'second'])). This can be - used for completer-specific commands such as reloading external - configuration. + used for completer-specific commands such as reloading external configuration. When the command is called with no arguments you should print a short summary of the supported commands or point the user to the help section where this - information can be found.""" + information can be found. + + Override the Shutdown() member function if your Completer subclass needs to do + custom cleanup logic on server shutdown.""" __metaclass__ = abc.ABCMeta - def __init__( self ): - self.triggers_for_filetype = TriggersForFiletype() + def __init__( self, user_options ): + self.user_options = user_options + self.min_num_chars = user_options[ 'min_num_of_chars_for_completion' ] + self.triggers_for_filetype = TriggersForFiletype( + user_options[ 'semantic_triggers' ] ) self.completions_future = None - self.completions_cache = None - self.completion_start_column = None + self._completions_cache = None # It's highly likely you DON'T want to override this function but the *Inner # version of it. - def ShouldUseNow( self, start_column ): - inner_says_yes = self.ShouldUseNowInner( start_column ) + def ShouldUseNow( self, request_data ): + inner_says_yes = self.ShouldUseNowInner( request_data ) if not inner_says_yes: - self.completions_cache = None + self._completions_cache = None - previous_results_were_empty = ( self.completions_cache and - self.completions_cache.CacheValid( - start_column ) and - not self.completions_cache.raw_completions ) + previous_results_were_empty = ( self._completions_cache and + self._completions_cache.CacheValid( + request_data[ 'line_num' ], + request_data[ 'start_column' ] ) and + not self._completions_cache.raw_completions ) return inner_says_yes and not previous_results_were_empty - def ShouldUseNowInner( self, start_column ): - line = vim.current.line - line_length = len( line ) + def ShouldUseNowInner( self, request_data ): + current_line = request_data[ 'line_value' ] + start_column = request_data[ 'start_column' ] + line_length = len( current_line ) if not line_length or start_column - 1 >= line_length: return False - filetype = self._CurrentFiletype() + filetype = self._CurrentFiletype( request_data[ 'filetypes' ] ) triggers = self.triggers_for_filetype[ filetype ] for trigger in triggers: @@ -151,7 +131,7 @@ class Completer( object ): trigger_length = len( trigger ) while True: line_index = start_column + index - if line_index < 0 or line[ line_index ] != trigger[ index ]: + if line_index < 0 or current_line[ line_index ] != trigger[ index ]: break if abs( index ) == trigger_length: @@ -160,154 +140,113 @@ class Completer( object ): return False - def QueryLengthAboveMinThreshold( self, start_column ): - query_length = vimsupport.CurrentColumn() - start_column - return query_length >= MIN_NUM_CHARS + def QueryLengthAboveMinThreshold( self, request_data ): + query_length = request_data[ 'column_num' ] - request_data[ 'start_column' ] + return query_length >= self.min_num_chars + # It's highly likely you DON'T want to override this function but the *Inner # version of it. - def CandidatesForQueryAsync( self, query, start_column ): - self.completion_start_column = start_column + def ComputeCandidates( self, request_data ): + if ( not ForceSemanticCompletion( request_data ) and + not self.ShouldUseNow( request_data ) ): + return [] - if query and self.completions_cache and self.completions_cache.CacheValid( - start_column ): - self.completions_cache.filtered_completions = ( - self.FilterAndSortCandidates( - self.completions_cache.raw_completions, - query ) ) + candidates = self._GetCandidatesFromSubclass( request_data ) + if request_data[ 'query' ]: + candidates = self.FilterAndSortCandidates( candidates, + request_data[ 'query' ] ) + return candidates + + + def _GetCandidatesFromSubclass( self, request_data ): + if ( self._completions_cache and + self._completions_cache.CacheValid( request_data[ 'line_num' ], + request_data[ 'start_column' ] ) ): + return self._completions_cache.raw_completions else: - self.completions_cache = None - self.CandidatesForQueryAsyncInner( query, start_column ) + self._completions_cache = CompletionsCache() + self._completions_cache.raw_completions = self.ComputeCandidatesInner( + request_data ) + self._completions_cache.line = request_data[ 'line_num' ] + self._completions_cache.column = request_data[ 'start_column' ] + return self._completions_cache.raw_completions + + + def ComputeCandidatesInner( self, request_data ): + pass def DefinedSubcommands( self ): return [] - def EchoUserCommandsHelpMessage( self ): + def UserCommandsHelpMessage( self ): subcommands = self.DefinedSubcommands() if subcommands: - vimsupport.EchoText( 'Supported commands are:\n' + - '\n'.join( subcommands ) + - '\nSee the docs for information on what they do.' ) + return ( 'Supported commands are:\n' + + '\n'.join( subcommands ) + + '\nSee the docs for information on what they do.' ) else: - vimsupport.EchoText( 'No supported subcommands' ) + return 'This Completer has no supported subcommands.' def FilterAndSortCandidates( self, candidates, query ): if not candidates: return [] - if hasattr( candidates, 'words' ): - candidates = candidates.words - items_are_objects = 'word' in candidates[ 0 ] + # We need to handle both an omni_completer style completer and a server + # style completer + if 'words' in candidates: + candidates = candidates[ 'words' ] - matches = ycm_core.FilterAndSortCandidates( + sort_property = '' + if 'word' in candidates[ 0 ]: + sort_property = 'word' + elif 'insertion_text' in candidates[ 0 ]: + sort_property = 'insertion_text' + + matches = ycm_client_support.FilterAndSortCandidates( candidates, - 'word' if items_are_objects else '', - query ) + sort_property, + ToUtf8IfNeeded( query ) ) return matches - def CandidatesForQueryAsyncInner( self, query, start_column ): + def OnFileReadyToParse( self, request_data ): pass - # It's highly likely you DON'T want to override this function but the *Inner - # version of it. - def AsyncCandidateRequestReady( self ): - if self.completions_cache: - return True - else: - return self.AsyncCandidateRequestReadyInner() - - - def AsyncCandidateRequestReadyInner( self ): - if not self.completions_future: - # We return True so that the caller can extract the default value from the - # future - return True - return self.completions_future.ResultsReady() - - - # It's highly likely you DON'T want to override this function but the *Inner - # version of it. - def CandidatesFromStoredRequest( self ): - if self.completions_cache: - return self.completions_cache.filtered_completions - else: - self.completions_cache = CompletionsCache() - self.completions_cache.raw_completions = self.CandidatesFromStoredRequestInner() - self.completions_cache.line, _ = vimsupport.CurrentLineAndColumn() - self.completions_cache.column = self.completion_start_column - return self.completions_cache.raw_completions - - - def CandidatesFromStoredRequestInner( self ): - if not self.completions_future: - return [] - return self.completions_future.GetResults() - - - def OnFileReadyToParse( self ): + def OnBufferVisit( self, request_data ): pass - def OnCursorMovedInsertMode( self ): + def OnBufferUnload( self, request_data ): pass - def OnCursorMovedNormalMode( self ): + def OnInsertLeave( self, request_data ): pass - def OnBufferVisit( self ): + def OnUserCommand( self, arguments, request_data ): + raise NotImplementedError( NO_USER_COMMANDS ) + + + def OnCurrentIdentifierFinished( self, request_data ): pass - def OnBufferUnload( self, deleted_buffer_file ): - pass - - - def OnCursorHold( self ): - pass - - - def OnInsertLeave( self ): - pass - - - def OnVimLeave( self ): - pass - - - def OnUserCommand( self, arguments ): - vimsupport.PostVimMessage( NO_USER_COMMANDS ) - - - def OnCurrentIdentifierFinished( self ): - pass - - - def DiagnosticsForCurrentFileReady( self ): - return False - - - def GetDiagnosticsForCurrentFile( self ): + def GetDiagnosticsForCurrentFile( self, request_data ): return [] - def ShowDetailedDiagnostic( self ): + def GetDetailedDiagnostic( self, request_data ): pass - def GettingCompletions( self ): - return False - - - def _CurrentFiletype( self ): - filetypes = vimsupport.CurrentFiletypes() + def _CurrentFiletype( self, filetypes ): supported = self.SupportedFiletypes() for filetype in filetypes: @@ -322,10 +261,14 @@ class Completer( object ): pass - def DebugInfo( self ): + def DebugInfo( self, request_data ): return '' + def Shutdown( self ): + pass + + class CompletionsCache( object ): def __init__( self ): self.line = -1 @@ -334,9 +277,7 @@ class CompletionsCache( object ): self.filtered_completions = [] - def CacheValid( self, start_column ): - completion_line, _ = vimsupport.CurrentLineAndColumn() - completion_column = start_column - return completion_line == self.line and completion_column == self.column + def CacheValid( self, current_line, start_column ): + return current_line == self.line and start_column == self.column diff --git a/python/ycm/completers/completer_utils.py b/python/ycm/completers/completer_utils.py index 1f4d0e50..c700e690 100644 --- a/python/ycm/completers/completer_utils.py +++ b/python/ycm/completers/completer_utils.py @@ -19,7 +19,7 @@ from collections import defaultdict from copy import deepcopy -import vim +import os DEFAULT_FILETYPE_TRIGGERS = { 'c' : ['->', '.'], @@ -58,12 +58,21 @@ def _FiletypeDictUnion( dict_one, dict_two ): return final_dict -def TriggersForFiletype(): - user_triggers = _FiletypeTriggerDictFromSpec( - vim.eval( 'g:ycm_semantic_triggers' ) ) - +def TriggersForFiletype( user_triggers ): default_triggers = _FiletypeTriggerDictFromSpec( DEFAULT_FILETYPE_TRIGGERS ) - return _FiletypeDictUnion( default_triggers, user_triggers ) + return _FiletypeDictUnion( default_triggers, dict( user_triggers ) ) + +def _PathToCompletersFolder(): + dir_of_current_script = os.path.dirname( os.path.abspath( __file__ ) ) + return os.path.join( dir_of_current_script ) + + +def PathToFiletypeCompleterPluginLoader( filetype ): + return os.path.join( _PathToCompletersFolder(), filetype, 'hook.py' ) + + +def FiletypeCompleterExistsForFiletype( filetype ): + return os.path.exists( PathToFiletypeCompleterPluginLoader( filetype ) ) diff --git a/python/ycm/completers/cpp/clang_completer.py b/python/ycm/completers/cpp/clang_completer.py index dfbb1ab9..fb034cee 100644 --- a/python/ycm/completers/cpp/clang_completer.py +++ b/python/ycm/completers/cpp/clang_completer.py @@ -18,116 +18,84 @@ # along with YouCompleteMe. If not, see . from collections import defaultdict -import vim import ycm_core -from ycm import vimsupport +from ycm.server import responses from ycm import extra_conf_store +from ycm.utils import ToUtf8IfNeeded from ycm.completers.completer import Completer -from ycm.completers.cpp.flags import Flags +from ycm.completers.cpp.flags import Flags, PrepareFlagsForClang CLANG_FILETYPES = set( [ 'c', 'cpp', 'objc', 'objcpp' ] ) -MAX_DIAGNOSTICS_TO_DISPLAY = int( vimsupport.GetVariableValue( - "g:ycm_max_diagnostics_to_display" ) ) +MIN_LINES_IN_FILE_TO_PARSE = 5 +PARSING_FILE_MESSAGE = 'Still parsing file, no completions yet.' +NO_COMPILE_FLAGS_MESSAGE = 'Still no compile flags, no completions yet.' +INVALID_FILE_MESSAGE = 'File is invalid.' +NO_COMPLETIONS_MESSAGE = 'No completions found; errors in the file?' +FILE_TOO_SHORT_MESSAGE = ( + 'File is less than {0} lines long; not compiling.'.format( + MIN_LINES_IN_FILE_TO_PARSE ) ) +NO_DIAGNOSTIC_MESSAGE = 'No diagnostic for current line!' class ClangCompleter( Completer ): - def __init__( self ): - super( ClangCompleter, self ).__init__() - self.completer = ycm_core.ClangCompleter() - self.completer.EnableThreading() - self.contents_holder = [] - self.filename_holder = [] - self.last_prepared_diagnostics = [] - self.parse_future = None - self.flags = Flags() - self.diagnostic_store = None - - # We set this flag when a compilation request comes in while one is already - # in progress. We use this to trigger the pending request after the previous - # one completes (from GetDiagnosticsForCurrentFile because that's the only - # method that knows when the compilation has finished). - self.extra_parse_desired = False + def __init__( self, user_options ): + super( ClangCompleter, self ).__init__( user_options ) + self._max_diagnostics_to_display = user_options[ + 'max_diagnostics_to_display' ] + self._completer = ycm_core.ClangCompleter() + self._flags = Flags() + self._diagnostic_store = None def SupportedFiletypes( self ): return CLANG_FILETYPES - def GetUnsavedFilesVector( self ): - # CAREFUL HERE! For UnsavedFile filename and contents we are referring - # directly to Python-allocated and -managed memory since we are accepting - # pointers to data members of python objects. We need to ensure that those - # objects outlive our UnsavedFile objects. This is why we need the - # contents_holder and filename_holder lists, to make sure the string objects - # are still around when we call CandidatesForQueryAndLocationInFile. We do - # this to avoid an extra copy of the entire file contents. - + def GetUnsavedFilesVector( self, request_data ): files = ycm_core.UnsavedFileVec() - self.contents_holder = [] - self.filename_holder = [] - for buffer in vimsupport.GetUnsavedBuffers(): - if not ClangAvailableForBuffer( buffer ): + for filename, file_data in request_data[ 'file_data' ].iteritems(): + if not ClangAvailableForFiletypes( file_data[ 'filetypes' ] ): continue - contents = '\n'.join( buffer ) - name = buffer.name - if not contents or not name: + contents = file_data[ 'contents' ] + if not contents or not filename: continue - self.contents_holder.append( contents ) - self.filename_holder.append( name ) unsaved_file = ycm_core.UnsavedFile() - unsaved_file.contents_ = self.contents_holder[ -1 ] - unsaved_file.length_ = len( self.contents_holder[ -1 ] ) - unsaved_file.filename_ = self.filename_holder[ -1 ] + utf8_contents = ToUtf8IfNeeded( contents ) + unsaved_file.contents_ = utf8_contents + unsaved_file.length_ = len( utf8_contents ) + unsaved_file.filename_ = ToUtf8IfNeeded( filename ) files.append( unsaved_file ) - return files - def CandidatesForQueryAsync( self, query, start_column ): - filename = vim.current.buffer.name - + def ComputeCandidatesInner( self, request_data ): + filename = request_data[ 'filepath' ] if not filename: return - if self.completer.UpdatingTranslationUnit( filename ): - vimsupport.PostVimMessage( 'Still parsing file, no completions yet.' ) - self.completions_future = None - return + if self._completer.UpdatingTranslationUnit( ToUtf8IfNeeded( filename ) ): + raise RuntimeError( PARSING_FILE_MESSAGE ) - flags = self.flags.FlagsForFile( filename ) + flags = self._FlagsForRequest( request_data ) if not flags: - vimsupport.PostVimMessage( 'Still no compile flags, no completions yet.' ) - self.completions_future = None - return + raise RuntimeError( NO_COMPILE_FLAGS_MESSAGE ) - # TODO: sanitize query, probably in C++ code - - files = ycm_core.UnsavedFileVec() - if not query: - files = self.GetUnsavedFilesVector() - - line, _ = vim.current.window.cursor - column = start_column + 1 - self.completions_future = ( - self.completer.CandidatesForQueryAndLocationInFileAsync( - query, - filename, + files = self.GetUnsavedFilesVector( request_data ) + line = request_data[ 'line_num' ] + 1 + column = request_data[ 'start_column' ] + 1 + results = self._completer.CandidatesForLocationInFile( + ToUtf8IfNeeded( filename ), line, column, files, - flags ) ) + flags ) - - def CandidatesFromStoredRequest( self ): - if not self.completions_future: - return [] - results = [ CompletionDataToDict( x ) for x in - self.completions_future.GetResults() ] if not results: - vimsupport.PostVimMessage( 'No completions found; errors in the file?' ) - return results + raise RuntimeError( NO_COMPLETIONS_MESSAGE ) + + return [ ConvertCompletionData( x ) for x in results ] def DefinedSubcommands( self ): @@ -137,157 +105,118 @@ class ClangCompleter( Completer ): 'ClearCompilationFlagCache'] - def OnUserCommand( self, arguments ): + def OnUserCommand( self, arguments, request_data ): if not arguments: - self.EchoUserCommandsHelpMessage() - return + raise ValueError( self.UserCommandsHelpMessage() ) command = arguments[ 0 ] if command == 'GoToDefinition': - self._GoToDefinition() + return self._GoToDefinition( request_data ) elif command == 'GoToDeclaration': - self._GoToDeclaration() + return self._GoToDeclaration( request_data ) elif command == 'GoToDefinitionElseDeclaration': - self._GoToDefinitionElseDeclaration() + return self._GoToDefinitionElseDeclaration( request_data ) elif command == 'ClearCompilationFlagCache': - self._ClearCompilationFlagCache() + return self._ClearCompilationFlagCache() + raise ValueError( self.UserCommandsHelpMessage() ) - def _LocationForGoTo( self, goto_function ): - filename = vim.current.buffer.name + def _LocationForGoTo( self, goto_function, request_data ): + filename = request_data[ 'filepath' ] if not filename: - return None + raise ValueError( INVALID_FILE_MESSAGE ) - flags = self.flags.FlagsForFile( filename ) + flags = self._FlagsForRequest( request_data ) if not flags: - vimsupport.PostVimMessage( 'Still no compile flags, can\'t compile.' ) - return None + raise ValueError( NO_COMPILE_FLAGS_MESSAGE ) - files = self.GetUnsavedFilesVector() - line, column = vimsupport.CurrentLineAndColumn() - # Making the line & column 1-based instead of 0-based - line += 1 - column += 1 - return getattr( self.completer, goto_function )( - filename, + files = self.GetUnsavedFilesVector( request_data ) + line = request_data[ 'line_num' ] + 1 + column = request_data[ 'column_num' ] + 1 + return getattr( self._completer, goto_function )( + ToUtf8IfNeeded( filename ), line, column, files, flags ) - def _GoToDefinition( self ): - location = self._LocationForGoTo( 'GetDefinitionLocation' ) + def _GoToDefinition( self, request_data ): + location = self._LocationForGoTo( 'GetDefinitionLocation', request_data ) if not location or not location.IsValid(): - vimsupport.PostVimMessage( 'Can\'t jump to definition.' ) - return + raise RuntimeError( 'Can\'t jump to definition.' ) - vimsupport.JumpToLocation( location.filename_, - location.line_number_, - location.column_number_ ) + return responses.BuildGoToResponse( location.filename_, + location.line_number_ - 1, + location.column_number_ - 1) - def _GoToDeclaration( self ): - location = self._LocationForGoTo( 'GetDeclarationLocation' ) + def _GoToDeclaration( self, request_data ): + location = self._LocationForGoTo( 'GetDeclarationLocation', request_data ) if not location or not location.IsValid(): - vimsupport.PostVimMessage( 'Can\'t jump to declaration.' ) - return + raise RuntimeError( 'Can\'t jump to declaration.' ) - vimsupport.JumpToLocation( location.filename_, - location.line_number_, - location.column_number_ ) + return responses.BuildGoToResponse( location.filename_, + location.line_number_ - 1, + location.column_number_ - 1) - def _GoToDefinitionElseDeclaration( self ): - location = self._LocationForGoTo( 'GetDefinitionLocation' ) + def _GoToDefinitionElseDeclaration( self, request_data ): + location = self._LocationForGoTo( 'GetDefinitionLocation', request_data ) if not location or not location.IsValid(): - location = self._LocationForGoTo( 'GetDeclarationLocation' ) + location = self._LocationForGoTo( 'GetDeclarationLocation', request_data ) if not location or not location.IsValid(): - vimsupport.PostVimMessage( 'Can\'t jump to definition or declaration.' ) - return + raise RuntimeError( 'Can\'t jump to definition or declaration.' ) + + return responses.BuildGoToResponse( location.filename_, + location.line_number_ - 1, + location.column_number_ - 1) - vimsupport.JumpToLocation( location.filename_, - location.line_number_, - location.column_number_ ) def _ClearCompilationFlagCache( self ): - self.flags.Clear() + self._flags.Clear() + def OnFileReadyToParse( self, request_data ): + filename = request_data[ 'filepath' ] + contents = request_data[ 'file_data' ][ filename ][ 'contents' ] + if contents.count( '\n' ) < MIN_LINES_IN_FILE_TO_PARSE: + raise ValueError( FILE_TOO_SHORT_MESSAGE ) - def OnFileReadyToParse( self ): - if vimsupport.NumLinesInBuffer( vim.current.buffer ) < 5: - self.parse_future = None - return - - filename = vim.current.buffer.name if not filename: - return + raise ValueError( INVALID_FILE_MESSAGE ) - if self.completer.UpdatingTranslationUnit( filename ): - self.extra_parse_desired = True - return - - flags = self.flags.FlagsForFile( filename ) + flags = self._FlagsForRequest( request_data ) if not flags: - self.parse_future = None - return + raise ValueError( NO_COMPILE_FLAGS_MESSAGE ) - self.parse_future = self.completer.UpdateTranslationUnitAsync( - filename, - self.GetUnsavedFilesVector(), + diagnostics = self._completer.UpdateTranslationUnit( + ToUtf8IfNeeded( filename ), + self.GetUnsavedFilesVector( request_data ), flags ) - self.extra_parse_desired = False + self._diagnostic_store = DiagnosticsToDiagStructure( diagnostics ) + return [ ConvertToDiagnosticResponse( x ) for x in + diagnostics[ : self._max_diagnostics_to_display ] ] - def OnBufferUnload( self, deleted_buffer_file ): - self.completer.DeleteCachesForFileAsync( deleted_buffer_file ) + def OnBufferUnload( self, request_data ): + self._completer.DeleteCachesForFile( + ToUtf8IfNeeded( request_data[ 'unloaded_buffer' ] ) ) - def DiagnosticsForCurrentFileReady( self ): - if not self.parse_future: - return False + def GetDetailedDiagnostic( self, request_data ): + current_line = request_data[ 'line_num' ] + 1 + current_column = request_data[ 'column_num' ] + 1 + current_file = request_data[ 'filepath' ] - return self.parse_future.ResultsReady() + if not self._diagnostic_store: + raise ValueError( NO_DIAGNOSTIC_MESSAGE ) - - def GettingCompletions( self ): - return self.completer.UpdatingTranslationUnit( vim.current.buffer.name ) - - - def GetDiagnosticsForCurrentFile( self ): - if self.DiagnosticsForCurrentFileReady(): - diagnostics = self.completer.DiagnosticsForFile( vim.current.buffer.name ) - self.diagnostic_store = DiagnosticsToDiagStructure( diagnostics ) - self.last_prepared_diagnostics = [ DiagnosticToDict( x ) for x in - diagnostics[ : MAX_DIAGNOSTICS_TO_DISPLAY ] ] - self.parse_future = None - - if self.extra_parse_desired: - self.OnFileReadyToParse() - - return self.last_prepared_diagnostics - - - def ShowDetailedDiagnostic( self ): - current_line, current_column = vimsupport.CurrentLineAndColumn() - - # CurrentLineAndColumn() numbers are 0-based, clang numbers are 1-based - current_line += 1 - current_column += 1 - - current_file = vim.current.buffer.name - - if not self.diagnostic_store: - vimsupport.PostVimMessage( "No diagnostic for current line!" ) - return - - diagnostics = self.diagnostic_store[ current_file ][ current_line ] + diagnostics = self._diagnostic_store[ current_file ][ current_line ] if not diagnostics: - vimsupport.PostVimMessage( "No diagnostic for current line!" ) - return + raise ValueError( NO_DIAGNOSTIC_MESSAGE ) closest_diagnostic = None distance_to_closest_diagnostic = 999 @@ -298,50 +227,36 @@ class ClangCompleter( Completer ): distance_to_closest_diagnostic = distance closest_diagnostic = diagnostic - vimsupport.EchoText( closest_diagnostic.long_formatted_text_ ) + return responses.BuildDisplayMessageResponse( + closest_diagnostic.long_formatted_text_ ) - def ShouldUseNow( self, start_column ): - # We don't want to use the Completer API cache, we use one in the C++ code. - return self.ShouldUseNowInner( start_column ) - - - def DebugInfo( self ): - filename = vim.current.buffer.name + def DebugInfo( self, request_data ): + filename = request_data[ 'filepath' ] if not filename: return '' - flags = self.flags.FlagsForFile( filename ) or [] + flags = self._FlagsForRequest( request_data ) or [] source = extra_conf_store.ModuleFileForSourceFile( filename ) return 'Flags for {0} loaded from {1}:\n{2}'.format( filename, source, list( flags ) ) -# TODO: make these functions module-local -def CompletionDataToDict( completion_data ): - # see :h complete-items for a description of the dictionary fields - return { - 'word' : completion_data.TextToInsertInBuffer(), - 'abbr' : completion_data.MainCompletionText(), - 'menu' : completion_data.ExtraMenuInfo(), - 'kind' : completion_data.kind_, - 'info' : completion_data.DetailedInfoForPreviewWindow(), - 'dup' : 1, - } + def _FlagsForRequest( self, request_data ): + filename = request_data[ 'filepath' ] + if 'compilation_flags' in request_data: + return PrepareFlagsForClang( request_data[ 'compilation_flags' ], + filename ) + return self._flags.FlagsForFile( filename ) -def DiagnosticToDict( diagnostic ): - # see :h getqflist for a description of the dictionary fields - return { - # TODO: wrap the bufnr generation into a function - 'bufnr' : int( vim.eval( "bufnr('{0}', 1)".format( - diagnostic.filename_ ) ) ), - 'lnum' : diagnostic.line_number_, - 'col' : diagnostic.column_number_, - 'text' : diagnostic.text_, - 'type' : diagnostic.kind_, - 'valid' : 1 - } +def ConvertCompletionData( completion_data ): + return responses.BuildCompletionData( + insertion_text = completion_data.TextToInsertInBuffer(), + menu_text = completion_data.MainCompletionText(), + extra_menu_info = completion_data.ExtraMenuInfo(), + kind = completion_data.kind_, + detailed_info = completion_data.DetailedInfoForPreviewWindow() ) def DiagnosticsToDiagStructure( diagnostics ): @@ -352,12 +267,19 @@ def DiagnosticsToDiagStructure( diagnostics ): return structure -def ClangAvailableForBuffer( buffer_object ): - filetypes = vimsupport.FiletypesForBuffer( buffer_object ) +def ClangAvailableForFiletypes( filetypes ): return any( [ filetype in CLANG_FILETYPES for filetype in filetypes ] ) -def InCFamilyFile(): - return any( [ filetype in CLANG_FILETYPES for filetype in - vimsupport.CurrentFiletypes() ] ) +def InCFamilyFile( filetypes ): + return ClangAvailableForFiletypes( filetypes ) + + +def ConvertToDiagnosticResponse( diagnostic ): + return responses.BuildDiagnosticData( diagnostic.filename_, + diagnostic.line_number_ - 1, + diagnostic.column_number_ - 1, + diagnostic.text_, + diagnostic.kind_ ) + diff --git a/python/ycm/completers/cpp/flags.py b/python/ycm/completers/cpp/flags.py index 61a241f9..385bdc03 100644 --- a/python/ycm/completers/cpp/flags.py +++ b/python/ycm/completers/cpp/flags.py @@ -19,12 +19,12 @@ import ycm_core import os -from ycm import vimsupport from ycm import extra_conf_store +from ycm.utils import ToUtf8IfNeeded -NO_EXTRA_CONF_FILENAME_MESSAGE = ('No {0} file detected, so no compile flags ' +NO_EXTRA_CONF_FILENAME_MESSAGE = ( 'No {0} file detected, so no compile flags ' 'are available. Thus no semantic support for C/C++/ObjC/ObjC++. Go READ THE ' - 'DOCS *NOW*, DON\'T file a bug report.').format( + 'DOCS *NOW*, DON\'T file a bug report.' ).format( extra_conf_store.YCM_EXTRA_CONF_FILENAME ) @@ -47,8 +47,8 @@ class Flags( object ): module = extra_conf_store.ModuleForSourceFile( filename ) if not module: if not self.no_extra_conf_file_warning_posted: - vimsupport.PostVimMessage( NO_EXTRA_CONF_FILENAME_MESSAGE ) self.no_extra_conf_file_warning_posted = True + raise RuntimeError( NO_EXTRA_CONF_FILENAME_MESSAGE ) return None results = module.FlagsForFile( filename ) @@ -59,7 +59,7 @@ class Flags( object ): flags = list( results[ 'flags' ] ) if add_special_clang_flags: flags += self.special_clang_flags - sanitized_flags = _PrepareFlagsForClang( flags, filename ) + sanitized_flags = PrepareFlagsForClang( flags, filename ) if results[ 'do_cache' ]: self.flags_for_file[ filename ] = sanitized_flags @@ -95,7 +95,7 @@ class Flags( object ): self.flags_for_file.clear() -def _PrepareFlagsForClang( flags, filename ): +def PrepareFlagsForClang( flags, filename ): flags = _RemoveUnusedFlags( flags, filename ) flags = _SanitizeFlags( flags ) return flags @@ -121,7 +121,7 @@ def _SanitizeFlags( flags ): vector = ycm_core.StringVec() for flag in sanitized_flags: - vector.append( flag ) + vector.append( ToUtf8IfNeeded( flag ) ) return vector diff --git a/python/ycm/completers/cpp/hook.py b/python/ycm/completers/cpp/hook.py index a4b02e0b..7c6f20e5 100644 --- a/python/ycm/completers/cpp/hook.py +++ b/python/ycm/completers/cpp/hook.py @@ -20,9 +20,9 @@ import ycm_core from ycm.completers.cpp.clang_completer import ClangCompleter -def GetCompleter(): +def GetCompleter( user_options ): if ycm_core.HasClangSupport(): - return ClangCompleter() + return ClangCompleter( user_options ) else: return None diff --git a/python/ycm/completers/cs/cs_completer.py b/python/ycm/completers/cs/cs_completer.py index afabffaa..4bf2f527 100755 --- a/python/ycm/completers/cs/cs_completer.py +++ b/python/ycm/completers/cs/cs_completer.py @@ -18,39 +18,37 @@ # You should have received a copy of the GNU General Public License # along with YouCompleteMe. If not, see . -import vim import os from sys import platform import glob -from ycm.completers.threaded_completer import ThreadedCompleter -from ycm import vimsupport +from ycm.completers.completer import Completer +from ycm.server import responses +from ycm import utils import urllib2 import urllib import urlparse import json import subprocess -import tempfile - +import logging SERVER_NOT_FOUND_MSG = ( 'OmniSharp server binary not found at {0}. ' + 'Did you compile it? You can do so by running ' + '"./install.sh --omnisharp-completer".' ) -class CsharpCompleter( ThreadedCompleter ): + +class CsharpCompleter( Completer ): """ A Completer that uses the Omnisharp server as completion engine. """ - def __init__( self ): - super( CsharpCompleter, self ).__init__() + def __init__( self, user_options ): + super( CsharpCompleter, self ).__init__( user_options ) self._omnisharp_port = None - - if vimsupport.GetBoolValue( 'g:ycm_auto_start_csharp_server' ): - self._StartServer() + self._logger = logging.getLogger( __name__ ) - def OnVimLeave( self ): - if ( vimsupport.GetBoolValue( 'g:ycm_auto_stop_csharp_server' ) and + def Shutdown( self ): + if ( self.user_options[ 'auto_start_csharp_server' ] and self._ServerIsRunning() ): self._StopServer() @@ -60,79 +58,83 @@ class CsharpCompleter( ThreadedCompleter ): return [ 'cs' ] - def ComputeCandidates( self, unused_query, unused_start_column ): - return [ { 'word': str( completion[ 'CompletionText' ] ), - 'menu': str( completion[ 'DisplayText' ] ), - 'info': str( completion[ 'Description' ] ) } - for completion in self._GetCompletions() ] + def ComputeCandidatesInner( self, request_data ): + return [ responses.BuildCompletionData( + completion[ 'CompletionText' ], + completion[ 'DisplayText' ], + completion[ 'Description' ] ) + for completion in self._GetCompletions( request_data ) ] def DefinedSubcommands( self ): return [ 'StartServer', 'StopServer', 'RestartServer', + 'ServerRunning', 'GoToDefinition', 'GoToDeclaration', 'GoToDefinitionElseDeclaration' ] - def OnUserCommand( self, arguments ): + def OnFileReadyToParse( self, request_data ): + if ( not self._omnisharp_port and + self.user_options[ 'auto_start_csharp_server' ] ): + self._StartServer( request_data ) + + + def OnUserCommand( self, arguments, request_data ): if not arguments: - self.EchoUserCommandsHelpMessage() - return + raise ValueError( self.UserCommandsHelpMessage() ) command = arguments[ 0 ] if command == 'StartServer': - self._StartServer() + self._StartServer( request_data ) elif command == 'StopServer': self._StopServer() elif command == 'RestartServer': if self._ServerIsRunning(): self._StopServer() - self._StartServer() + self._StartServer( request_data ) + elif command == 'ServerRunning': + return self._ServerIsRunning() elif command in [ 'GoToDefinition', 'GoToDeclaration', 'GoToDefinitionElseDeclaration' ]: - self._GoToDefinition() + return self._GoToDefinition( request_data ) + raise ValueError( self.UserCommandsHelpMessage() ) def DebugInfo( self ): if self._ServerIsRunning(): - return 'Server running at: {}\nLogfiles:\n{}\n{}'.format( - self._PortToHost(), self._filename_stdout, self._filename_stderr ) + return 'Server running at: {0}\nLogfiles:\n{1}\n{2}'.format( + self._ServerLocation(), self._filename_stdout, self._filename_stderr ) else: return 'Server is not running' - def _StartServer( self ): + def _StartServer( self, request_data ): """ Start the OmniSharp server """ - self._omnisharp_port = self._FindFreePort() - solutionfiles, folder = _FindSolutionFiles() + self._logger.info( 'startup' ) + + self._omnisharp_port = utils.GetUnusedLocalhostPort() + solutionfiles, folder = _FindSolutionFiles( request_data[ 'filepath' ] ) if len( solutionfiles ) == 0: - vimsupport.PostVimMessage( - 'Error starting OmniSharp server: no solutionfile found' ) - return + raise RuntimeError( + 'Error starting OmniSharp server: no solutionfile found' ) elif len( solutionfiles ) == 1: solutionfile = solutionfiles[ 0 ] else: - choice = vimsupport.PresentDialog( - 'Which solutionfile should be loaded?', - [ str( i ) + " " + solution for i, solution in - enumerate( solutionfiles ) ] ) - if choice == -1: - vimsupport.PostVimMessage( 'OmniSharp not started' ) - return - else: - solutionfile = solutionfiles[ choice ] + raise RuntimeError( + 'Found multiple solution files instead of one!\n{0}'.format( + solutionfiles ) ) omnisharp = os.path.join( os.path.abspath( os.path.dirname( __file__ ) ), 'OmniSharpServer/OmniSharp/bin/Debug/OmniSharp.exe' ) if not os.path.isfile( omnisharp ): - vimsupport.PostVimMessage( SERVER_NOT_FOUND_MSG.format( omnisharp ) ) - return + raise RuntimeError( SERVER_NOT_FOUND_MSG.format( omnisharp ) ) if not platform.startswith( 'win' ): omnisharp = 'mono ' + omnisharp @@ -142,8 +144,8 @@ class CsharpCompleter( ThreadedCompleter ): command = [ omnisharp + ' -p ' + str( self._omnisharp_port ) + ' -s ' + path_to_solutionfile ] - filename_format = ( tempfile.gettempdir() + - '/omnisharp_{port}_{sln}_{std}.log' ) + filename_format = os.path.join( utils.PathToTempDir(), + 'omnisharp_{port}_{sln}_{std}.log' ) self._filename_stdout = filename_format.format( port=self._omnisharp_port, sln=solutionfile, std='stdout' ) @@ -154,82 +156,72 @@ class CsharpCompleter( ThreadedCompleter ): with open( self._filename_stdout, 'w' ) as fstdout: subprocess.Popen( command, stdout=fstdout, stderr=fstderr, shell=True ) - vimsupport.PostVimMessage( 'Starting OmniSharp server' ) + self._logger.info( 'Starting OmniSharp server' ) def _StopServer( self ): """ Stop the OmniSharp server """ self._GetResponse( '/stopserver' ) self._omnisharp_port = None - vimsupport.PostVimMessage( 'Stopping OmniSharp server' ) + self._logger.info( 'Stopping OmniSharp server' ) - def _GetCompletions( self ): + def _GetCompletions( self, request_data ): """ Ask server for completions """ - completions = self._GetResponse( '/autocomplete', self._DefaultParameters() ) + completions = self._GetResponse( '/autocomplete', + self._DefaultParameters( request_data ) ) return completions if completions != None else [] - def _GoToDefinition( self ): + def _GoToDefinition( self, request_data ): """ Jump to definition of identifier under cursor """ - definition = self._GetResponse( '/gotodefinition', self._DefaultParameters() ) + definition = self._GetResponse( '/gotodefinition', + self._DefaultParameters( request_data ) ) if definition[ 'FileName' ] != None: - vimsupport.JumpToLocation( definition[ 'FileName' ], - definition[ 'Line' ], - definition[ 'Column' ] ) + return responses.BuildGoToResponse( definition[ 'FileName' ], + definition[ 'Line' ], + definition[ 'Column' ] ) else: - vimsupport.PostVimMessage( 'Can\'t jump to definition' ) + raise RuntimeError( 'Can\'t jump to definition' ) - def _DefaultParameters( self ): + def _DefaultParameters( self, request_data ): """ Some very common request parameters """ - line, column = vimsupport.CurrentLineAndColumn() parameters = {} - parameters[ 'line' ], parameters[ 'column' ] = line + 1, column + 1 - parameters[ 'buffer' ] = '\n'.join( vim.current.buffer ) - parameters[ 'filename' ] = vim.current.buffer.name + parameters[ 'line' ] = request_data[ 'line_num' ] + 1 + parameters[ 'column' ] = request_data[ 'column_num' ] + 1 + filepath = request_data[ 'filepath' ] + parameters[ 'buffer' ] = request_data[ 'file_data' ][ filepath ][ + 'contents' ] + parameters[ 'filename' ] = filepath return parameters def _ServerIsRunning( self ): """ Check if our OmniSharp server is running """ - return ( self._omnisharp_port != None and - self._GetResponse( '/checkalivestatus', silent=True ) != None ) - - - def _FindFreePort( self ): - """ Find port without an OmniSharp server running on it """ - port = int( vimsupport.GetVariableValue( - 'g:ycm_csharp_server_port' ) ) - while self._GetResponse( '/checkalivestatus', - silent=True, - port=port ) != None: - port += 1 - return port - - - def _PortToHost( self, port=None ): - if port == None: - port = self._omnisharp_port - return 'http://localhost:' + str( port ) - - - def _GetResponse( self, endPoint, parameters={}, silent=False, port=None ): - """ Handle communication with server """ - target = urlparse.urljoin( self._PortToHost( port ), endPoint ) - parameters = urllib.urlencode( parameters ) try: - response = urllib2.urlopen( target, parameters ) - return json.loads( response.read() ) - except Exception: - # TODO: Add logging for this case. We can't post a Vim message because Vim - # crashes when that's done from a no-GUI thread. - return None + return bool( self._omnisharp_port and + self._GetResponse( '/checkalivestatus', silent = True ) ) + except: + return False -def _FindSolutionFiles(): + def _ServerLocation( self ): + return 'http://localhost:' + str( self._omnisharp_port ) + + + def _GetResponse( self, handler, parameters = {}, silent = False ): + """ Handle communication with server """ + # TODO: Replace usage of urllib with Requests + target = urlparse.urljoin( self._ServerLocation(), handler ) + parameters = urllib.urlencode( parameters ) + response = urllib2.urlopen( target, parameters ) + return json.loads( response.read() ) + + +def _FindSolutionFiles( filepath ): """ Find solution files by searching upwards in the file tree """ - folder = os.path.dirname( vim.current.buffer.name ) + folder = os.path.dirname( filepath ) solutionfiles = glob.glob1( folder, '*.sln' ) while not solutionfiles: lastfolder = folder diff --git a/python/ycm/completers/cs/hook.py b/python/ycm/completers/cs/hook.py index d082bbcc..d266191c 100644 --- a/python/ycm/completers/cs/hook.py +++ b/python/ycm/completers/cs/hook.py @@ -17,5 +17,5 @@ from ycm.completers.cs.cs_completer import CsharpCompleter -def GetCompleter(): - return CsharpCompleter() +def GetCompleter( user_options ): + return CsharpCompleter( user_options ) diff --git a/python/ycm/completers/general/filename_completer.py b/python/ycm/completers/general/filename_completer.py index 8dd1f181..87f10bd7 100644 --- a/python/ycm/completers/general/filename_completer.py +++ b/python/ycm/completers/general/filename_completer.py @@ -16,26 +16,21 @@ # You should have received a copy of the GNU General Public License # along with YouCompleteMe. If not, see . -import vim import os import re -from ycm import vimsupport -from ycm.completers.threaded_completer import ThreadedCompleter +from ycm.completers.completer import Completer from ycm.completers.cpp.clang_completer import InCFamilyFile from ycm.completers.cpp.flags import Flags +from ycm.server import responses -USE_WORKING_DIR = vimsupport.GetBoolValue( - 'g:ycm_filepath_completion_use_working_dir' ) - - -class FilenameCompleter( ThreadedCompleter ): +class FilenameCompleter( Completer ): """ General completer that provides filename and filepath completions. """ - def __init__( self ): - super( FilenameCompleter, self ).__init__() + def __init__( self, user_options ): + super( FilenameCompleter, self ).__init__( user_options ) self._flags = Flags() self._path_regex = re.compile( """ @@ -57,25 +52,35 @@ class FilenameCompleter( ThreadedCompleter ): self._include_regex = re.compile( include_regex_common ) - def AtIncludeStatementStart( self, start_column ): - return ( InCFamilyFile() and + def AtIncludeStatementStart( self, request_data ): + start_column = request_data[ 'start_column' ] + current_line = request_data[ 'line_value' ] + filepath = request_data[ 'filepath' ] + filetypes = request_data[ 'file_data' ][ filepath ][ 'filetypes' ] + return ( InCFamilyFile( filetypes ) and self._include_start_regex.match( - vim.current.line[ :start_column ] ) ) + current_line[ :start_column ] ) ) - def ShouldUseNowInner( self, start_column ): - return ( start_column and ( vim.current.line[ start_column - 1 ] == '/' or - self.AtIncludeStatementStart( start_column ) ) ) + def ShouldUseNowInner( self, request_data ): + start_column = request_data[ 'start_column' ] + current_line = request_data[ 'line_value' ] + return ( start_column and ( current_line[ start_column - 1 ] == '/' or + self.AtIncludeStatementStart( request_data ) ) ) def SupportedFiletypes( self ): return [] - def ComputeCandidates( self, unused_query, start_column ): - line = vim.current.line[ :start_column ] + def ComputeCandidatesInner( self, request_data ): + current_line = request_data[ 'line_value' ] + start_column = request_data[ 'start_column' ] + filepath = request_data[ 'filepath' ] + filetypes = request_data[ 'file_data' ][ filepath ][ 'filetypes' ] + line = current_line[ :start_column ] - if InCFamilyFile(): + if InCFamilyFile( filetypes ): include_match = self._include_regex.search( line ) if include_match: path_dir = line[ include_match.end(): ] @@ -83,20 +88,26 @@ class FilenameCompleter( ThreadedCompleter ): # http://gcc.gnu.org/onlinedocs/cpp/Include-Syntax.html include_current_file_dir = '<' not in include_match.group() return _GenerateCandidatesForPaths( - self.GetPathsIncludeCase( path_dir, include_current_file_dir ) ) + self.GetPathsIncludeCase( path_dir, + include_current_file_dir, + filepath ) ) path_match = self._path_regex.search( line ) path_dir = os.path.expanduser( path_match.group() ) if path_match else '' - return _GenerateCandidatesForPaths( _GetPathsStandardCase( path_dir ) ) + return _GenerateCandidatesForPaths( + _GetPathsStandardCase( + path_dir, + self.user_options[ 'filepath_completion_use_working_dir' ], + filepath ) ) - def GetPathsIncludeCase( self, path_dir, include_current_file_dir ): + def GetPathsIncludeCase( self, path_dir, include_current_file_dir, filepath ): paths = [] - include_paths = self._flags.UserIncludePaths( vim.current.buffer.name ) + include_paths = self._flags.UserIncludePaths( filepath ) if include_current_file_dir: - include_paths.append( os.path.dirname( vim.current.buffer.name ) ) + include_paths.append( os.path.dirname( filepath ) ) for include_path in include_paths: try: @@ -110,9 +121,9 @@ class FilenameCompleter( ThreadedCompleter ): return sorted( set( paths ) ) -def _GetPathsStandardCase( path_dir ): - if not USE_WORKING_DIR and not path_dir.startswith( '/' ): - path_dir = os.path.join( os.path.dirname( vim.current.buffer.name ), +def _GetPathsStandardCase( path_dir, use_working_dir, filepath ): + if not use_working_dir and not path_dir.startswith( '/' ): + path_dir = os.path.join( os.path.dirname( filepath ), path_dir ) try: @@ -135,8 +146,8 @@ def _GenerateCandidatesForPaths( absolute_paths ): seen_basenames.add( basename ) is_dir = os.path.isdir( absolute_path ) - completion_dicts.append( { 'word': basename, - 'dup': 1, - 'menu': '[Dir]' if is_dir else '[File]' } ) + completion_dicts.append( + responses.BuildCompletionData( basename, + '[Dir]' if is_dir else '[File]' ) ) return completion_dicts diff --git a/python/ycm/completers/general/general_completer_store.py b/python/ycm/completers/general/general_completer_store.py index c04972ae..f47dff8d 100644 --- a/python/ycm/completers/general/general_completer_store.py +++ b/python/ycm/completers/general/general_completer_store.py @@ -21,13 +21,7 @@ from ycm.completers.completer import Completer from ycm.completers.all.identifier_completer import IdentifierCompleter from ycm.completers.general.filename_completer import FilenameCompleter - -try: - from ycm.completers.general.ultisnips_completer import UltiSnipsCompleter - USE_ULTISNIPS_COMPLETER = True -except ImportError: - USE_ULTISNIPS_COMPLETER = False - +from ycm.completers.general.ultisnips_completer import UltiSnipsCompleter class GeneralCompleterStore( Completer ): @@ -38,12 +32,11 @@ class GeneralCompleterStore( Completer ): GeneralCompleterStore are passed to all general completers. """ - def __init__( self ): - super( GeneralCompleterStore, self ).__init__() - self._identifier_completer = IdentifierCompleter() - self._filename_completer = FilenameCompleter() - self._ultisnips_completer = ( UltiSnipsCompleter() - if USE_ULTISNIPS_COMPLETER else None ) + def __init__( self, user_options ): + super( GeneralCompleterStore, self ).__init__( user_options ) + self._identifier_completer = IdentifierCompleter( user_options ) + self._filename_completer = FilenameCompleter( user_options ) + self._ultisnips_completer = UltiSnipsCompleter( user_options ) self._non_filename_completers = filter( lambda x: x, [ self._ultisnips_completer, self._identifier_completer ] ) @@ -58,17 +51,21 @@ class GeneralCompleterStore( Completer ): return set() - def ShouldUseNow( self, start_column ): + def GetIdentifierCompleter( self ): + return self._identifier_completer + + + def ShouldUseNow( self, request_data ): self._current_query_completers = [] - if self._filename_completer.ShouldUseNow( start_column ): + if self._filename_completer.ShouldUseNow( request_data ): self._current_query_completers = [ self._filename_completer ] return True should_use_now = False for completer in self._non_filename_completers: - should_use_this_completer = completer.ShouldUseNow( start_column ) + should_use_this_completer = completer.ShouldUseNow( request_data ) should_use_now = should_use_now or should_use_this_completer if should_use_this_completer: @@ -77,69 +74,49 @@ class GeneralCompleterStore( Completer ): return should_use_now - def CandidatesForQueryAsync( self, query, start_column ): - for completer in self._current_query_completers: - completer.CandidatesForQueryAsync( query, start_column ) + def ComputeCandidates( self, request_data ): + if not self.ShouldUseNow( request_data ): + return [] - - def AsyncCandidateRequestReady( self ): - return all( x.AsyncCandidateRequestReady() for x in - self._current_query_completers ) - - - def CandidatesFromStoredRequest( self ): candidates = [] for completer in self._current_query_completers: - candidates += completer.CandidatesFromStoredRequest() + candidates += completer.ComputeCandidates( request_data ) return candidates - def OnFileReadyToParse( self ): + def OnFileReadyToParse( self, request_data ): for completer in self._all_completers: - completer.OnFileReadyToParse() + completer.OnFileReadyToParse( request_data ) - def OnCursorMovedInsertMode( self ): + def OnBufferVisit( self, request_data ): for completer in self._all_completers: - completer.OnCursorMovedInsertMode() + completer.OnBufferVisit( request_data ) - def OnCursorMovedNormalMode( self ): + def OnBufferUnload( self, request_data ): for completer in self._all_completers: - completer.OnCursorMovedNormalMode() + completer.OnBufferUnload( request_data ) - def OnBufferVisit( self ): + def OnInsertLeave( self, request_data ): for completer in self._all_completers: - completer.OnBufferVisit() + completer.OnInsertLeave( request_data ) - def OnBufferUnload( self, deleted_buffer_file ): + def OnCurrentIdentifierFinished( self, request_data ): for completer in self._all_completers: - completer.OnBufferUnload( deleted_buffer_file ) - - - def OnCursorHold( self ): - for completer in self._all_completers: - completer.OnCursorHold() - - - def OnInsertLeave( self ): - for completer in self._all_completers: - completer.OnInsertLeave() - - - def OnVimLeave( self ): - for completer in self._all_completers: - completer.OnVimLeave() - - - def OnCurrentIdentifierFinished( self ): - for completer in self._all_completers: - completer.OnCurrentIdentifierFinished() + completer.OnCurrentIdentifierFinished( request_data ) def GettingCompletions( self ): for completer in self._all_completers: completer.GettingCompletions() + + + def Shutdown( self ): + for completer in self._all_completers: + completer.Shutdown() + + diff --git a/python/ycm/completers/general/ultisnips_completer.py b/python/ycm/completers/general/ultisnips_completer.py index 32710a10..ca7b666e 100644 --- a/python/ycm/completers/general/ultisnips_completer.py +++ b/python/ycm/completers/general/ultisnips_completer.py @@ -19,7 +19,7 @@ # along with YouCompleteMe. If not, see . from ycm.completers.general_completer import GeneralCompleter -from UltiSnips import UltiSnips_Manager +from ycm.server import responses class UltiSnipsCompleter( GeneralCompleter ): @@ -27,42 +27,28 @@ class UltiSnipsCompleter( GeneralCompleter ): General completer that provides UltiSnips snippet names in completions. """ - def __init__( self ): - super( UltiSnipsCompleter, self ).__init__() + def __init__( self, user_options ): + super( UltiSnipsCompleter, self ).__init__( user_options ) self._candidates = None self._filtered_candidates = None - def ShouldUseNowInner( self, start_column ): - return self.QueryLengthAboveMinThreshold( start_column ) + def ShouldUseNow( self, request_data ): + return self.QueryLengthAboveMinThreshold( request_data ) - def CandidatesForQueryAsync( self, query, unused_start_column ): - self._filtered_candidates = self.FilterAndSortCandidates( self._candidates, - query ) + def ComputeCandidates( self, request_data ): + if not self.ShouldUseNow( request_data ): + return [] + return self.FilterAndSortCandidates( + self._candidates, request_data[ 'query' ] ) - def AsyncCandidateRequestReady( self ): - return True + def OnBufferVisit( self, request_data ): + raw_candidates = request_data.get( 'ultisnips_snippets', [] ) + self._candidates = [ + responses.BuildCompletionData( + str( snip[ 'trigger' ] ), + str( ' ' + snip[ 'description' ].encode( 'utf-8' ) ) ) + for snip in raw_candidates ] - - def CandidatesFromStoredRequest( self ): - return self._filtered_candidates if self._filtered_candidates else [] - - - def OnBufferVisit( self ): - self._candidates = _GetCandidates() - - -def _GetCandidates(): - try: - rawsnips = UltiSnips_Manager._snips( '', 1 ) - - # UltiSnips_Manager._snips() returns a class instance where: - # class.trigger - name of snippet trigger word ( e.g. defn or testcase ) - # class.description - description of the snippet - return [ { 'word': str( snip.trigger ), - 'menu': str( ' ' + snip.description.encode('utf-8') ) } - for snip in rawsnips ] - except: - return [] diff --git a/python/ycm/completers/general_completer.py b/python/ycm/completers/general_completer.py index dd59e1b6..71f013e0 100644 --- a/python/ycm/completers/general_completer.py +++ b/python/ycm/completers/general_completer.py @@ -29,8 +29,8 @@ class GeneralCompleter( Completer ): Subclass Completer directly. """ - def __init__( self ): - super( GeneralCompleter, self ).__init__() + def __init__( self, user_options ): + super( GeneralCompleter, self ).__init__( user_options ) def SupportedFiletypes( self ): diff --git a/python/ycm/completers/objc/hook.py b/python/ycm/completers/objc/hook.py index 737f795c..a26da7ed 100644 --- a/python/ycm/completers/objc/hook.py +++ b/python/ycm/completers/objc/hook.py @@ -20,8 +20,8 @@ import ycm_core from ycm.completers.cpp.clang_completer import ClangCompleter -def GetCompleter(): +def GetCompleter( user_options ): if ycm_core.HasClangSupport(): - return ClangCompleter() + return ClangCompleter( user_options ) else: return None diff --git a/python/ycm/completers/objcpp/hook.py b/python/ycm/completers/objcpp/hook.py index a4b02e0b..7c6f20e5 100644 --- a/python/ycm/completers/objcpp/hook.py +++ b/python/ycm/completers/objcpp/hook.py @@ -20,9 +20,9 @@ import ycm_core from ycm.completers.cpp.clang_completer import ClangCompleter -def GetCompleter(): +def GetCompleter( user_options ): if ycm_core.HasClangSupport(): - return ClangCompleter() + return ClangCompleter( user_options ) else: return None diff --git a/python/ycm/completers/python/hook.py b/python/ycm/completers/python/hook.py index db8ac883..690b7923 100644 --- a/python/ycm/completers/python/hook.py +++ b/python/ycm/completers/python/hook.py @@ -17,5 +17,5 @@ from ycm.completers.python.jedi_completer import JediCompleter -def GetCompleter(): - return JediCompleter() +def GetCompleter( user_options ): + return JediCompleter( user_options ) diff --git a/python/ycm/completers/python/jedi b/python/ycm/completers/python/jedi deleted file mode 160000 index d5d12716..00000000 --- a/python/ycm/completers/python/jedi +++ /dev/null @@ -1 +0,0 @@ -Subproject commit d5d12716b1d67df9cbaa4d3ea0c90e47c0023208 diff --git a/python/ycm/completers/python/jedi_completer.py b/python/ycm/completers/python/jedi_completer.py index 5d454a06..158b4171 100644 --- a/python/ycm/completers/python/jedi_completer.py +++ b/python/ycm/completers/python/jedi_completer.py @@ -19,34 +19,25 @@ # You should have received a copy of the GNU General Public License # along with YouCompleteMe. If not, see . -import vim -from ycm.completers.threaded_completer import ThreadedCompleter -from ycm import vimsupport +from ycm.completers.completer import Completer +from ycm.server import responses -import sys -from os.path import join, abspath, dirname - -# We need to add the jedi package to sys.path, but it's important that we clean -# up after ourselves, because ycm.YouCompletMe.GetFiletypeCompleterForFiletype -# removes sys.path[0] after importing completers.python.hook -sys.path.insert( 0, join( abspath( dirname( __file__ ) ), 'jedi' ) ) try: import jedi except ImportError: - vimsupport.PostVimMessage( + raise ImportError( 'Error importing jedi. Make sure the jedi submodule has been checked out. ' 'In the YouCompleteMe folder, run "git submodule update --init --recursive"') -sys.path.pop( 0 ) -class JediCompleter( ThreadedCompleter ): +class JediCompleter( Completer ): """ A Completer that uses the Jedi completion engine. https://jedi.readthedocs.org/en/latest/ """ - def __init__( self ): - super( JediCompleter, self ).__init__() + def __init__( self, user_options ): + super( JediCompleter, self ).__init__( user_options ) def SupportedFiletypes( self ): @@ -54,113 +45,109 @@ class JediCompleter( ThreadedCompleter ): return [ 'python' ] - def _GetJediScript( self ): - contents = '\n'.join( vim.current.buffer ) - line, column = vimsupport.CurrentLineAndColumn() + def _GetJediScript( self, request_data ): + filename = request_data[ 'filepath' ] + contents = request_data[ 'file_data' ][ filename ][ 'contents' ] # Jedi expects lines to start at 1, not 0 - line += 1 - filename = vim.current.buffer.name + line = request_data[ 'line_num' ] + 1 + column = request_data[ 'column_num' ] return jedi.Script( contents, line, column, filename ) - def ComputeCandidates( self, unused_query, unused_start_column ): - script = self._GetJediScript() - - return [ { 'word': str( completion.name ), - 'menu': str( completion.description ), - 'info': str( completion.doc ) } + def ComputeCandidatesInner( self, request_data ): + script = self._GetJediScript( request_data ) + return [ responses.BuildCompletionData( + str( completion.name ), + str( completion.description ), + str( completion.doc ) ) for completion in script.completions() ] - def DefinedSubcommands( self ): - return [ "GoToDefinition", - "GoToDeclaration", - "GoToDefinitionElseDeclaration" ] + return [ 'GoToDefinition', + 'GoToDeclaration', + 'GoToDefinitionElseDeclaration' ] - def OnUserCommand( self, arguments ): + def OnUserCommand( self, arguments, request_data ): if not arguments: - self.EchoUserCommandsHelpMessage() - return + raise ValueError( self.UserCommandsHelpMessage() ) command = arguments[ 0 ] if command == 'GoToDefinition': - self._GoToDefinition() + return self._GoToDefinition( request_data ) elif command == 'GoToDeclaration': - self._GoToDeclaration() + return self._GoToDeclaration( request_data ) elif command == 'GoToDefinitionElseDeclaration': - self._GoToDefinitionElseDeclaration() + return self._GoToDefinitionElseDeclaration( request_data ) + raise ValueError( self.UserCommandsHelpMessage() ) - def _GoToDefinition( self ): - definitions = self._GetDefinitionsList() + def _GoToDefinition( self, request_data ): + definitions = self._GetDefinitionsList( request_data ) if definitions: - self._JumpToLocation( definitions ) + return self._BuildGoToResponse( definitions ) else: - vimsupport.PostVimMessage( 'Can\'t jump to definition.' ) + raise RuntimeError( 'Can\'t jump to definition.' ) - def _GoToDeclaration( self ): - definitions = self._GetDefinitionsList( declaration = True ) + def _GoToDeclaration( self, request_data ): + definitions = self._GetDefinitionsList( request_data, declaration = True ) if definitions: - self._JumpToLocation( definitions ) + return self._BuildGoToResponse( definitions ) else: - vimsupport.PostVimMessage( 'Can\'t jump to declaration.' ) + raise RuntimeError( 'Can\'t jump to declaration.' ) - def _GoToDefinitionElseDeclaration( self ): - definitions = self._GetDefinitionsList() or \ - self._GetDefinitionsList( declaration = True ) + def _GoToDefinitionElseDeclaration( self, request_data ): + definitions = ( self._GetDefinitionsList( request_data ) or + self._GetDefinitionsList( request_data, declaration = True ) ) if definitions: - self._JumpToLocation( definitions ) + return self._BuildGoToResponse( definitions ) else: - vimsupport.PostVimMessage( 'Can\'t jump to definition or declaration.' ) + raise RuntimeError( 'Can\'t jump to definition or declaration.' ) - def _GetDefinitionsList( self, declaration = False ): + def _GetDefinitionsList( self, request_data, declaration = False ): definitions = [] - script = self._GetJediScript() + script = self._GetJediScript( request_data ) try: if declaration: definitions = script.goto_definitions() else: definitions = script.goto_assignments() except jedi.NotFoundError: - vimsupport.PostVimMessage( - "Cannot follow nothing. Put your cursor on a valid name." ) - except Exception as e: - vimsupport.PostVimMessage( - "Caught exception, aborting. Full error: " + str( e ) ) + raise RuntimeError( + 'Cannot follow nothing. Put your cursor on a valid name.' ) return definitions - def _JumpToLocation( self, definition_list ): + def _BuildGoToResponse( self, definition_list ): if len( definition_list ) == 1: definition = definition_list[ 0 ] if definition.in_builtin_module(): if definition.is_keyword: - vimsupport.PostVimMessage( - "Cannot get the definition of Python keywords." ) + raise RuntimeError( + 'Cannot get the definition of Python keywords.' ) else: - vimsupport.PostVimMessage( "Builtin modules cannot be displayed." ) + raise RuntimeError( 'Builtin modules cannot be displayed.' ) else: - vimsupport.JumpToLocation( definition.module_path, - definition.line, - definition.column + 1 ) + return responses.BuildGoToResponse( definition.module_path, + definition.line - 1, + definition.column ) else: # multiple definitions defs = [] for definition in definition_list: if definition.in_builtin_module(): - defs.append( {'text': 'Builtin ' + \ - definition.description.encode( 'utf-8' ) } ) + defs.append( responses.BuildDescriptionOnlyGoToResponse( + 'Builtin ' + definition.description ) ) else: - defs.append( {'filename': definition.module_path.encode( 'utf-8' ), - 'lnum': definition.line, - 'col': definition.column + 1, - 'text': definition.description.encode( 'utf-8' ) } ) + defs.append( + responses.BuildGoToResponse( definition.module_path, + definition.line - 1, + definition.column, + definition.description ) ) + return defs - vim.eval( 'setqflist( %s )' % repr( defs ) ) - vim.eval( 'youcompleteme#OpenGoToList()' ) diff --git a/python/ycm/completers/threaded_completer.py b/python/ycm/completers/threaded_completer.py deleted file mode 100644 index fd6192aa..00000000 --- a/python/ycm/completers/threaded_completer.py +++ /dev/null @@ -1,102 +0,0 @@ -#!/usr/bin/env python -# -# Copyright (C) 2011, 2012 Strahinja Val Markovic -# -# This file is part of YouCompleteMe. -# -# YouCompleteMe is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# YouCompleteMe is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with YouCompleteMe. If not, see . - -import abc -from threading import Thread, Event -from ycm.completers.completer import Completer - -class ThreadedCompleter( Completer ): - """A subclass of Completer that abstracts away the use of a background thread. - - This is a great class to subclass for your custom completer. It will provide - you with async computation of candidates when you implement the - ComputeCandidates() method; no need to worry about threads, locks, events or - similar. - - If you use this class as the base class for your Completer, you DON'T need to - provide implementations for the CandidatesForQueryAsync(), - AsyncCandidateRequestReady() and CandidatesFromStoredRequest() functions. Just - implement ComputeCandidates(). - - For examples of subclasses of this class, see the following files: - python/completers/general/filename_completer.py - """ - - def __init__( self ): - super( ThreadedCompleter, self ).__init__() - self._query_ready = Event() - self._candidates_ready = Event() - self._candidates = None - self._start_completion_thread() - - - def _start_completion_thread( self ): - self._completion_thread = Thread( target=self.SetCandidates ) - self._completion_thread.daemon = True - self._completion_thread.start() - - - def CandidatesForQueryAsyncInner( self, query, start_column ): - self._candidates = None - self._candidates_ready.clear() - self._query = query - self._start_column = start_column - self._query_ready.set() - - - def AsyncCandidateRequestReadyInner( self ): - return WaitAndClearIfSet( self._candidates_ready, timeout=0.005 ) - - - def CandidatesFromStoredRequestInner( self ): - return self._candidates or [] - - - @abc.abstractmethod - def ComputeCandidates( self, query, start_column ): - """This function should compute the candidates to show to the user. - The return value should be of the same type as that for - CandidatesFromStoredRequest().""" - pass - - - def SetCandidates( self ): - while True: - try: - WaitAndClearIfSet( self._query_ready ) - self._candidates = self.ComputeCandidates( self._query, - self._start_column ) - except: - self._query_ready.clear() - self._candidates = [] - self._candidates_ready.set() - - -def WaitAndClearIfSet( event, timeout=None ): - """Given an |event| and a |timeout|, waits for the event a maximum of timeout - seconds. After waiting, clears the event if it's set and returns the state of - the event before it was cleared.""" - - # We can't just do flag_is_set = event.wait( timeout ) because that breaks on - # Python 2.6 - event.wait( timeout ) - flag_is_set = event.is_set() - if flag_is_set: - event.clear() - return flag_is_set diff --git a/python/ycm/extra_conf_store.py b/python/ycm/extra_conf_store.py index 698dac2e..d49576c4 100644 --- a/python/ycm/extra_conf_store.py +++ b/python/ycm/extra_conf_store.py @@ -24,25 +24,30 @@ import imp import random import string import sys -import vim -from ycm import vimsupport +import logging +from threading import Lock +from ycm import user_options_store +from ycm.server.responses import UnknownExtraConf from fnmatch import fnmatch # Constants YCM_EXTRA_CONF_FILENAME = '.ycm_extra_conf.py' -CONFIRM_CONF_FILE_MESSAGE = ('Found {0}. Load? \n\n(Question can be turned ' - 'off with options, see YCM docs)') -GLOBAL_YCM_EXTRA_CONF_FILE = os.path.expanduser( - vimsupport.GetVariableValue( "g:ycm_global_ycm_extra_conf" ) -) # Singleton variables _module_for_module_file = {} +_module_for_module_file_lock = Lock() _module_file_for_source_file = {} +_module_file_for_source_file_lock = Lock() + + +def Reset(): + global _module_for_module_file, _module_file_for_source_file + _module_for_module_file = {} + _module_file_for_source_file = {} def ModuleForSourceFile( filename ): - return _Load( ModuleFileForSourceFile( filename ) ) + return Load( ModuleFileForSourceFile( filename ) ) def ModuleFileForSourceFile( filename ): @@ -50,35 +55,50 @@ def ModuleFileForSourceFile( filename ): order and return the filename of the first module that was allowed to load. If no module was found or allowed to load, None is returned.""" - if not filename in _module_file_for_source_file: - for module_file in _ExtraConfModuleSourceFilesForFile( filename ): - if _Load( module_file ): - _module_file_for_source_file[ filename ] = module_file - break + with _module_file_for_source_file_lock: + if not filename in _module_file_for_source_file: + for module_file in _ExtraConfModuleSourceFilesForFile( filename ): + if Load( module_file ): + _module_file_for_source_file[ filename ] = module_file + break return _module_file_for_source_file.setdefault( filename ) -def CallExtraConfYcmCorePreloadIfExists(): - _CallExtraConfMethod( 'YcmCorePreload' ) +def CallGlobalExtraConfYcmCorePreloadIfExists(): + _CallGlobalExtraConfMethod( 'YcmCorePreload' ) -def CallExtraConfVimCloseIfExists(): - _CallExtraConfMethod( 'VimClose' ) +def Shutdown(): + # VimClose is for the sake of backwards compatibility; it's a no-op when it + # doesn't exist. + _CallGlobalExtraConfMethod( 'VimClose' ) + _CallGlobalExtraConfMethod( 'Shutdown' ) -def _CallExtraConfMethod( function_name ): - vim_current_working_directory = vim.eval( 'getcwd()' ) - path_to_dummy = os.path.join( vim_current_working_directory, 'DUMMY_FILE' ) - module = ModuleForSourceFile( path_to_dummy ) - if not module or not hasattr( module, function_name ): +def _CallGlobalExtraConfMethod( function_name ): + logger = _Logger() + global_ycm_extra_conf = _GlobalYcmExtraConfFileLocation() + if not ( global_ycm_extra_conf and + os.path.exists( global_ycm_extra_conf ) ): + logger.debug( 'No global extra conf, not calling method ' + function_name ) return + + module = Load( global_ycm_extra_conf, force = True ) + if not module or not hasattr( module, function_name ): + logger.debug( 'Global extra conf not loaded or no function ' + + function_name ) + return + + logger.info( 'Calling global extra conf method {0} on conf file {1}'.format( + function_name, global_ycm_extra_conf ) ) getattr( module, function_name )() def _Disable( module_file ): """Disables the loading of a module for the current session.""" - _module_for_module_file[ module_file ] = None + with _module_for_module_file_lock: + _module_for_module_file[ module_file ] = None def _ShouldLoad( module_file ): @@ -86,20 +106,25 @@ def _ShouldLoad( module_file ): decide using a white-/blacklist and ask the user for confirmation as a fallback.""" - if ( module_file == GLOBAL_YCM_EXTRA_CONF_FILE or - not vimsupport.GetBoolValue( 'g:ycm_confirm_extra_conf' ) ): + if ( module_file == _GlobalYcmExtraConfFileLocation() or + not user_options_store.Value( 'confirm_extra_conf' ) ): return True - globlist = vimsupport.GetVariableValue( 'g:ycm_extra_conf_globlist' ) + globlist = user_options_store.Value( 'extra_conf_globlist' ) for glob in globlist: is_blacklisted = glob[0] == '!' if _MatchesGlobPattern( module_file, glob.lstrip('!') ): return not is_blacklisted - return vimsupport.Confirm( CONFIRM_CONF_FILE_MESSAGE.format( module_file ) ) + # We disable the file if it's unknown so that we don't ask the user about it + # repeatedly. Raising UnknownExtraConf should result in the client sending + # another request to load the module file if the user explicitly chooses to do + # that. + _Disable( module_file ) + raise UnknownExtraConf( module_file ) -def _Load( module_file, force = False ): +def Load( module_file, force = False ): """Load and return the module contained in a file. Using force = True the module will be loaded regardless of the criteria in _ShouldLoad. @@ -109,11 +134,13 @@ def _Load( module_file, force = False ): return None if not force: - if module_file in _module_for_module_file: - return _module_for_module_file[ module_file ] + with _module_for_module_file_lock: + if module_file in _module_for_module_file: + return _module_for_module_file[ module_file ] if not _ShouldLoad( module_file ): - return _Disable( module_file ) + _Disable( module_file ) + return None # This has to be here because a long time ago, the ycm_extra_conf.py files # used to import clang_helpers.py from the cpp folder. This is not needed @@ -123,7 +150,8 @@ def _Load( module_file, force = False ): module = imp.load_source( _RandomName(), module_file ) del sys.path[ 0 ] - _module_for_module_file[ module_file ] = module + with _module_for_module_file_lock: + _module_for_module_file[ module_file ] = module return module @@ -139,15 +167,16 @@ def _MatchesGlobPattern( filename, glob ): def _ExtraConfModuleSourceFilesForFile( filename ): """For a given filename, search all parent folders for YCM_EXTRA_CONF_FILENAME files that will compute the flags necessary to compile the file. - If GLOBAL_YCM_EXTRA_CONF_FILE exists it is returned as a fallback.""" + If _GlobalYcmExtraConfFileLocation() exists it is returned as a fallback.""" for folder in _PathsToAllParentFolders( filename ): candidate = os.path.join( folder, YCM_EXTRA_CONF_FILENAME ) if os.path.exists( candidate ): yield candidate - if ( GLOBAL_YCM_EXTRA_CONF_FILE - and os.path.exists( GLOBAL_YCM_EXTRA_CONF_FILE ) ): - yield GLOBAL_YCM_EXTRA_CONF_FILE + global_ycm_extra_conf = _GlobalYcmExtraConfFileLocation() + if ( global_ycm_extra_conf + and os.path.exists( global_ycm_extra_conf ) ): + yield global_ycm_extra_conf def _PathsToAllParentFolders( filename ): @@ -188,3 +217,12 @@ def _DirectoryOfThisScript(): def _RandomName(): """Generates a random module name.""" return ''.join( random.choice( string.ascii_lowercase ) for x in range( 15 ) ) + + +def _GlobalYcmExtraConfFileLocation(): + return os.path.expanduser( + user_options_store.Value( 'global_ycm_extra_conf' ) ) + + +def _Logger(): + return logging.getLogger( __name__ ) diff --git a/python/ycm/server/__init__.py b/python/ycm/server/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/python/ycm/server/default_settings.json b/python/ycm/server/default_settings.json new file mode 100644 index 00000000..df4b3204 --- /dev/null +++ b/python/ycm/server/default_settings.json @@ -0,0 +1,32 @@ +{ + "filepath_completion_use_working_dir": 0, + "min_num_of_chars_for_completion": 2, + "min_num_identifier_candidate_chars": 0, + "semantic_triggers": {}, + "filetype_specific_completion_to_disable": { + "gitcommit": 1 + }, + "seed_identifiers_with_syntax": 0, + "collect_identifiers_from_comments_and_strings": 0, + "collect_identifiers_from_tags_files": 0, + "extra_conf_globlist": [], + "global_ycm_extra_conf": "", + "confirm_extra_conf": 1, + "complete_in_comments": 0, + "complete_in_strings": 1, + "max_diagnostics_to_display": 30, + "filetype_whitelist": { + "*": 1 + }, + "filetype_blacklist": { + "tagbar": 1, + "qf": 1, + "notes": 1, + "markdown": 1, + "unite": 1, + "text": 1 + }, + "auto_start_csharp_server": 1, + "auto_stop_csharp_server": 1, + "csharp_server_port": 2000 +} diff --git a/python/ycm/server/handlers.py b/python/ycm/server/handlers.py new file mode 100644 index 00000000..6640511a --- /dev/null +++ b/python/ycm/server/handlers.py @@ -0,0 +1,222 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +from os import path + +try: + import ycm_core +except ImportError as e: + raise RuntimeError( + 'Error importing ycm_core. Are you sure you have placed a ' + 'version 3.2+ libclang.[so|dll|dylib] in folder "{0}"? ' + 'See the Installation Guide in the docs. Full error: {1}'.format( + path.realpath( path.join( path.abspath( __file__ ), '../../..' ) ), + str( e ) ) ) + +import atexit +import logging +import json +import bottle +import httplib +from bottle import request, response +import server_state +from ycm import user_options_store +from ycm.server.responses import BuildExceptionResponse +from ycm import extra_conf_store + + +# num bytes for the request body buffer; request.json only works if the request +# size is less than this +bottle.Request.MEMFILE_MAX = 300 * 1024 + +# TODO: rename these to _lower_case +SERVER_STATE = None +LOGGER = logging.getLogger( __name__ ) +app = bottle.Bottle() + + +@app.post( '/event_notification' ) +def EventNotification(): + LOGGER.info( 'Received event notification' ) + request_data = request.json + event_name = request_data[ 'event_name' ] + LOGGER.debug( 'Event name: %s', event_name ) + + event_handler = 'On' + event_name + getattr( SERVER_STATE.GetGeneralCompleter(), event_handler )( request_data ) + + filetypes = request_data[ 'filetypes' ] + response_data = None + if SERVER_STATE.FiletypeCompletionUsable( filetypes ): + response_data = getattr( SERVER_STATE.GetFiletypeCompleter( filetypes ), + event_handler )( request_data ) + + if response_data: + return _JsonResponse( response_data ) + + +@app.post( '/run_completer_command' ) +def RunCompleterCommand(): + LOGGER.info( 'Received command request' ) + request_data = request.json + completer = _GetCompleterForRequestData( request_data ) + + return _JsonResponse( completer.OnUserCommand( + request_data[ 'command_arguments' ], + request_data ) ) + + +@app.post( '/completions' ) +def GetCompletions(): + LOGGER.info( 'Received completion request' ) + request_data = request.json + do_filetype_completion = SERVER_STATE.ShouldUseFiletypeCompleter( + request_data ) + LOGGER.debug( 'Using filetype completion: %s', do_filetype_completion ) + filetypes = request_data[ 'filetypes' ] + completer = ( SERVER_STATE.GetFiletypeCompleter( filetypes ) if + do_filetype_completion else + SERVER_STATE.GetGeneralCompleter() ) + + return _JsonResponse( completer.ComputeCandidates( request_data ) ) + + +@app.get( '/user_options' ) +def GetUserOptions(): + LOGGER.info( 'Received user options GET request' ) + return _JsonResponse( dict( SERVER_STATE.user_options ) ) + + +@app.get( '/healthy' ) +def GetHealthy(): + LOGGER.info( 'Received health request' ) + return _JsonResponse( True ) + + +@app.post( '/user_options' ) +def SetUserOptions(): + LOGGER.info( 'Received user options POST request' ) + UpdateUserOptions( request.json ) + + +@app.post( '/semantic_completion_available' ) +def FiletypeCompletionAvailable(): + LOGGER.info( 'Received filetype completion available request' ) + return _JsonResponse( SERVER_STATE.FiletypeCompletionAvailable( + request.json[ 'filetypes' ] ) ) + + +@app.post( '/defined_subcommands' ) +def DefinedSubcommands(): + LOGGER.info( 'Received defined subcommands request' ) + completer = _GetCompleterForRequestData( request.json ) + + return _JsonResponse( completer.DefinedSubcommands() ) + + +@app.post( '/detailed_diagnostic' ) +def GetDetailedDiagnostic(): + LOGGER.info( 'Received detailed diagnostic request' ) + request_data = request.json + completer = _GetCompleterForRequestData( request_data ) + + return _JsonResponse( completer.GetDetailedDiagnostic( request_data ) ) + + +@app.post( '/load_extra_conf_file' ) +def LoadExtraConfFile(): + LOGGER.info( 'Received extra conf load request' ) + request_data = request.json + extra_conf_store.Load( request_data[ 'filepath' ], force = True ) + + +@app.post( '/debug_info' ) +def DebugInfo(): + LOGGER.info( 'Received debug info request' ) + + output = [] + has_clang_support = ycm_core.HasClangSupport() + output.append( 'Server has Clang support compiled in: {0}'.format( + has_clang_support ) ) + + if has_clang_support: + output.append( 'Clang version: ' + ycm_core.ClangVersion() ) + + request_data = request.json + try: + output.append( + _GetCompleterForRequestData( request_data ).DebugInfo( request_data) ) + except: + pass + return _JsonResponse( '\n'.join( output ) ) + + +# The type of the param is Bottle.HTTPError +@app.error( httplib.INTERNAL_SERVER_ERROR ) +def ErrorHandler( httperror ): + return _JsonResponse( BuildExceptionResponse( httperror.exception, + httperror.traceback ) ) + + +def _JsonResponse( data ): + response.set_header( 'Content-Type', 'application/json' ) + return json.dumps( data, default = _UniversalSerialize ) + + +def _UniversalSerialize( obj ): + serialized = obj.__dict__.copy() + serialized[ 'TYPE' ] = type( obj ).__name__ + return serialized + + +def _GetCompleterForRequestData( request_data ): + completer_target = request_data.get( 'completer_target', None ) + + if completer_target == 'identifier': + return SERVER_STATE.GetGeneralCompleter().GetIdentifierCompleter() + elif completer_target == 'filetype_default' or not completer_target: + return SERVER_STATE.GetFiletypeCompleter( request_data[ 'filetypes' ] ) + else: + return SERVER_STATE.GetFiletypeCompleter( [ completer_target ] ) + + +@atexit.register +def ServerShutdown(): + LOGGER.info( 'Server shutting down' ) + if SERVER_STATE: + SERVER_STATE.Shutdown() + extra_conf_store.Shutdown() + + +def UpdateUserOptions( options ): + global SERVER_STATE + + if not options: + return + + user_options_store.SetAll( options ) + SERVER_STATE = server_state.ServerState( options ) + + +def SetServerStateToDefaults(): + global SERVER_STATE, LOGGER + LOGGER = logging.getLogger( __name__ ) + user_options_store.LoadDefaults() + SERVER_STATE = server_state.ServerState( user_options_store.GetAll() ) + extra_conf_store.Reset() diff --git a/python/ycm/server/responses.py b/python/ycm/server/responses.py new file mode 100644 index 00000000..f0ca6364 --- /dev/null +++ b/python/ycm/server/responses.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +import os + +CONFIRM_CONF_FILE_MESSAGE = ('Found {0}. Load? \n\n(Question can be turned ' + 'off with options, see YCM docs)') + +class ServerError( Exception ): + def __init__( self, message ): + super( ServerError, self ).__init__( message ) + + +class UnknownExtraConf( ServerError ): + def __init__( self, extra_conf_file ): + message = CONFIRM_CONF_FILE_MESSAGE.format( extra_conf_file ) + super( UnknownExtraConf, self ).__init__( message ) + self.extra_conf_file = extra_conf_file + + + +def BuildGoToResponse( filepath, line_num, column_num, description = None ): + response = { + 'filepath': os.path.realpath( filepath ), + 'line_num': line_num, + 'column_num': column_num + } + + if description: + response[ 'description' ] = description + return response + + +def BuildDescriptionOnlyGoToResponse( text ): + return { + 'description': text, + } + + +# TODO: Look at all the callers and ensure they are not using this instead of an +# exception. +def BuildDisplayMessageResponse( text ): + return { + 'message': text + } + + +def BuildCompletionData( insertion_text, + extra_menu_info = None, + detailed_info = None, + menu_text = None, + kind = None ): + completion_data = { + 'insertion_text': insertion_text + } + + if extra_menu_info: + completion_data[ 'extra_menu_info' ] = extra_menu_info + if menu_text: + completion_data[ 'menu_text' ] = menu_text + if detailed_info: + completion_data[ 'detailed_info' ] = detailed_info + if kind: + completion_data[ 'kind' ] = kind + return completion_data + + +def BuildDiagnosticData( filepath, + line_num, + column_num, + text, + kind ): + return { + 'filepath': filepath, + 'line_num': line_num, + 'column_num': column_num, + 'text': text, + 'kind': kind + } + + +def BuildExceptionResponse( exception, traceback ): + return { + 'exception': exception, + 'message': str( exception ), + 'traceback': traceback + } + diff --git a/python/ycm/server/server_state.py b/python/ycm/server/server_state.py new file mode 100644 index 00000000..39cd1243 --- /dev/null +++ b/python/ycm/server/server_state.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +import imp +import os +from ycm.utils import ForceSemanticCompletion +from ycm.completers.general.general_completer_store import GeneralCompleterStore +from ycm.completers.completer_utils import PathToFiletypeCompleterPluginLoader + + +class ServerState( object ): + def __init__( self, user_options ): + self._user_options = user_options + self._filetype_completers = {} + self._gencomp = GeneralCompleterStore( self._user_options ) + + + @property + def user_options( self ): + return self._user_options + + + def Shutdown( self ): + for completer in self._filetype_completers.itervalues(): + if completer: + completer.Shutdown() + + self._gencomp.Shutdown() + + + def _GetFiletypeCompleterForFiletype( self, filetype ): + try: + return self._filetype_completers[ filetype ] + except KeyError: + pass + + module_path = PathToFiletypeCompleterPluginLoader( filetype ) + completer = None + supported_filetypes = [ filetype ] + if os.path.exists( module_path ): + module = imp.load_source( filetype, module_path ) + completer = module.GetCompleter( self._user_options ) + if completer: + supported_filetypes.extend( completer.SupportedFiletypes() ) + + for supported_filetype in supported_filetypes: + self._filetype_completers[ supported_filetype ] = completer + return completer + + + def GetFiletypeCompleter( self, current_filetypes ): + completers = [ self._GetFiletypeCompleterForFiletype( filetype ) + for filetype in current_filetypes ] + + for completer in completers: + if completer: + return completer + + raise ValueError( 'No semantic completer exists for filetypes: {0}'.format( + current_filetypes ) ) + + + def FiletypeCompletionAvailable( self, filetypes ): + try: + self.GetFiletypeCompleter( filetypes ) + return True + except: + return False + + + def FiletypeCompletionUsable( self, filetypes ): + return ( self.CurrentFiletypeCompletionEnabled( filetypes ) and + self.FiletypeCompletionAvailable( filetypes ) ) + + + def ShouldUseGeneralCompleter( self, request_data ): + return self._gencomp.ShouldUseNow( request_data ) + + + def ShouldUseFiletypeCompleter( self, request_data ): + filetypes = request_data[ 'filetypes' ] + if self.FiletypeCompletionUsable( filetypes ): + return ( ForceSemanticCompletion( request_data ) or + self.GetFiletypeCompleter( filetypes ).ShouldUseNow( + request_data ) ) + return False + + + def GetGeneralCompleter( self ): + return self._gencomp + + + def CurrentFiletypeCompletionEnabled( self, current_filetypes ): + filetype_to_disable = self._user_options[ + 'filetype_specific_completion_to_disable' ] + return not all([ x in filetype_to_disable for x in current_filetypes ]) + diff --git a/python/ycm/server/server_utils.py b/python/ycm/server/server_utils.py new file mode 100644 index 00000000..96d566fe --- /dev/null +++ b/python/ycm/server/server_utils.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +import sys +import os + +def SetUpPythonPath(): + # We want to have the YouCompleteMe/python directory on the Python PATH + # because all the code already assumes that it's there. This is a relic from + # before the client/server architecture. + # TODO: Fix things so that this is not needed anymore when we split ycmd into + # a separate repository. + sys.path.insert( 0, os.path.join( + os.path.dirname( os.path.abspath( __file__ ) ), + '../..' ) ) + + from ycm import utils + utils.AddThirdPartyFoldersToSysPath() diff --git a/python/ycm/server/tests/__init__.py b/python/ycm/server/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/python/ycm/server/tests/basic_test.py b/python/ycm/server/tests/basic_test.py new file mode 100644 index 00000000..6d226324 --- /dev/null +++ b/python/ycm/server/tests/basic_test.py @@ -0,0 +1,451 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +import os +import httplib +import time +from ..server_utils import SetUpPythonPath +SetUpPythonPath() +from webtest import TestApp +from .. import handlers +from ..responses import BuildCompletionData, UnknownExtraConf +from nose.tools import ok_, eq_, with_setup +from hamcrest import ( assert_that, has_items, has_entry, contains, + contains_string, has_entries, contains_inanyorder ) +import bottle + +bottle.debug( True ) + +# TODO: Split this file into multiple files. + +def BuildRequest( **kwargs ): + filepath = kwargs[ 'filepath' ] if 'filepath' in kwargs else '/foo' + contents = kwargs[ 'contents' ] if 'contents' in kwargs else '' + filetype = kwargs[ 'filetype' ] if 'filetype' in kwargs else 'foo' + + request = { + 'query': '', + 'line_num': 0, + 'column_num': 0, + 'start_column': 0, + 'filetypes': [ filetype ], + 'filepath': filepath, + 'line_value': contents, + 'file_data': { + filepath: { + 'contents': contents, + 'filetypes': [ filetype ] + } + } + } + + for key, value in kwargs.iteritems(): + if key in [ 'contents', 'filetype', 'filepath' ]: + continue + request[ key ] = value + + if key == 'line_num': + lines = contents.splitlines() + if len( lines ) > 1: + # NOTE: assumes 0-based line_num + request[ 'line_value' ] = lines[ value ] + + return request + + +# TODO: Make the other tests use this helper too instead of BuildCompletionData +def CompletionEntryMatcher( insertion_text ): + return has_entry( 'insertion_text', insertion_text ) + + +def Setup(): + handlers.SetServerStateToDefaults() + + +def PathToTestDataDir(): + dir_of_current_script = os.path.dirname( os.path.abspath( __file__ ) ) + return os.path.join( dir_of_current_script, 'testdata' ) + + +def PathToTestFile( test_basename ): + return os.path.join( PathToTestDataDir(), test_basename ) + + +@with_setup( Setup ) +def GetCompletions_IdentifierCompleter_Works_test(): + app = TestApp( handlers.app ) + event_data = BuildRequest( contents = 'foo foogoo ba', + event_name = 'FileReadyToParse' ) + + app.post_json( '/event_notification', event_data ) + + completion_data = BuildRequest( contents = 'oo foo foogoo ba', + query = 'oo', + column_num = 2 ) + + eq_( [ BuildCompletionData( 'foo' ), + BuildCompletionData( 'foogoo' ) ], + app.post_json( '/completions', completion_data ).json ) + + +@with_setup( Setup ) +def GetCompletions_CsCompleter_Works_test(): + app = TestApp( handlers.app ) + filepath = PathToTestFile( 'testy/Program.cs' ) + contents = open( filepath ).read() + event_data = BuildRequest( filepath = filepath, + filetype = 'cs', + contents = contents, + event_name = 'FileReadyToParse' ) + + app.post_json( '/event_notification', event_data ) + + # We need to wait until the server has started up. + while True: + result = app.post_json( '/run_completer_command', + BuildRequest( completer_target = 'filetype_default', + command_arguments = ['ServerRunning'], + filetype = 'cs' ) ).json + if result: + break + time.sleep( 0.2 ) + + completion_data = BuildRequest( filepath = filepath, + filetype = 'cs', + contents = contents, + line_num = 8, + column_num = 11, + start_column = 11 ) + + results = app.post_json( '/completions', completion_data ).json + assert_that( results, has_items( CompletionEntryMatcher( 'CursorLeft' ), + CompletionEntryMatcher( 'CursorSize' ) ) ) + + +@with_setup( Setup ) +def GetCompletions_ClangCompleter_WorksWithExplicitFlags_test(): + app = TestApp( handlers.app ) + contents = """ +struct Foo { + int x; + int y; + char c; +}; + +int main() +{ + Foo foo; + foo. +} +""" + + # 0-based line and column! + completion_data = BuildRequest( filepath = '/foo.cpp', + filetype = 'cpp', + contents = contents, + line_num = 10, + column_num = 6, + start_column = 6, + compilation_flags = ['-x', 'c++'] ) + + results = app.post_json( '/completions', completion_data ).json + assert_that( results, has_items( CompletionEntryMatcher( 'c' ), + CompletionEntryMatcher( 'x' ), + CompletionEntryMatcher( 'y' ) ) ) + + +@with_setup( Setup ) +def GetCompletions_ClangCompleter_UnknownExtraConfException_test(): + app = TestApp( handlers.app ) + filepath = PathToTestFile( 'basic.cpp' ) + completion_data = BuildRequest( filepath = filepath, + filetype = 'cpp', + contents = open( filepath ).read(), + force_semantic = True ) + + response = app.post_json( '/completions', + completion_data, + expect_errors = True ) + + eq_( response.status_code, httplib.INTERNAL_SERVER_ERROR ) + assert_that( response.json, + has_entry( 'exception', + has_entry( 'TYPE', UnknownExtraConf.__name__ ) ) ) + + +@with_setup( Setup ) +def GetCompletions_ClangCompleter_WorksWhenExtraConfExplicitlyAllowed_test(): + app = TestApp( handlers.app ) + app.post_json( '/load_extra_conf_file', + { 'filepath': PathToTestFile( '.ycm_extra_conf.py' ) } ) + + filepath = PathToTestFile( 'basic.cpp' ) + completion_data = BuildRequest( filepath = filepath, + filetype = 'cpp', + contents = open( filepath ).read(), + line_num = 10, + column_num = 6, + start_column = 6 ) + + results = app.post_json( '/completions', completion_data ).json + assert_that( results, has_items( CompletionEntryMatcher( 'c' ), + CompletionEntryMatcher( 'x' ), + CompletionEntryMatcher( 'y' ) ) ) + + +@with_setup( Setup ) +def GetCompletions_ClangCompleter_ForceSemantic_OnlyFileteredCompletions_test(): + app = TestApp( handlers.app ) + contents = """ +int main() +{ + int foobar; + int floozar; + int gooboo; + int bleble; + + fooar +} +""" + + # 0-based line and column! + completion_data = BuildRequest( filepath = '/foo.cpp', + filetype = 'cpp', + force_semantic = True, + contents = contents, + line_num = 8, + column_num = 7, + start_column = 7, + query = 'fooar', + compilation_flags = ['-x', 'c++'] ) + + results = app.post_json( '/completions', completion_data ).json + assert_that( results, + contains_inanyorder( CompletionEntryMatcher( 'foobar' ), + CompletionEntryMatcher( 'floozar' ) ) ) + + +@with_setup( Setup ) +def GetCompletions_ForceSemantic_Works_test(): + app = TestApp( handlers.app ) + + completion_data = BuildRequest( filetype = 'python', + force_semantic = True ) + + results = app.post_json( '/completions', completion_data ).json + assert_that( results, has_items( CompletionEntryMatcher( 'abs' ), + CompletionEntryMatcher( 'open' ), + CompletionEntryMatcher( 'bool' ) ) ) + + +@with_setup( Setup ) +def GetCompletions_IdentifierCompleter_SyntaxKeywordsAdded_test(): + app = TestApp( handlers.app ) + event_data = BuildRequest( event_name = 'FileReadyToParse', + syntax_keywords = ['foo', 'bar', 'zoo'] ) + + app.post_json( '/event_notification', event_data ) + + completion_data = BuildRequest( contents = 'oo ', + query = 'oo', + column_num = 2 ) + + eq_( [ BuildCompletionData( 'foo' ), + BuildCompletionData( 'zoo' ) ], + app.post_json( '/completions', completion_data ).json ) + + +@with_setup( Setup ) +def GetCompletions_UltiSnipsCompleter_Works_test(): + app = TestApp( handlers.app ) + event_data = BuildRequest( + event_name = 'BufferVisit', + ultisnips_snippets = [ + {'trigger': 'foo', 'description': 'bar'}, + {'trigger': 'zoo', 'description': 'goo'}, + ] ) + + app.post_json( '/event_notification', event_data ) + + completion_data = BuildRequest( contents = 'oo ', + query = 'oo', + column_num = 2 ) + + eq_( [ BuildCompletionData( 'foo', ' bar' ), + BuildCompletionData( 'zoo', ' goo' ) ], + app.post_json( '/completions', completion_data ).json ) + + +@with_setup( Setup ) +def RunCompleterCommand_GoTo_Jedi_ZeroBasedLineAndColumn_test(): + app = TestApp( handlers.app ) + contents = """ +def foo(): + pass + +foo() +""" + + goto_data = BuildRequest( completer_target = 'filetype_default', + command_arguments = ['GoToDefinition'], + line_num = 4, + contents = contents, + filetype = 'python', + filepath = '/foo.py' ) + + # 0-based line and column! + eq_( { + 'filepath': '/foo.py', + 'line_num': 1, + 'column_num': 4 + }, + app.post_json( '/run_completer_command', goto_data ).json ) + + +@with_setup( Setup ) +def RunCompleterCommand_GoTo_Clang_ZeroBasedLineAndColumn_test(): + app = TestApp( handlers.app ) + contents = """ +struct Foo { + int x; + int y; + char c; +}; + +int main() +{ + Foo foo; + return 0; +} +""" + + goto_data = BuildRequest( completer_target = 'filetype_default', + command_arguments = ['GoToDefinition'], + compilation_flags = ['-x', 'c++'], + line_num = 9, + column_num = 2, + contents = contents, + filetype = 'cpp' ) + + # 0-based line and column! + eq_( { + 'filepath': '/foo', + 'line_num': 1, + 'column_num': 7 + }, + app.post_json( '/run_completer_command', goto_data ).json ) + + +@with_setup( Setup ) +def DefinedSubcommands_Works_test(): + app = TestApp( handlers.app ) + subcommands_data = BuildRequest( completer_target = 'python' ) + + eq_( [ 'GoToDefinition', + 'GoToDeclaration', + 'GoToDefinitionElseDeclaration' ], + app.post_json( '/defined_subcommands', subcommands_data ).json ) + + +@with_setup( Setup ) +def DefinedSubcommands_WorksWhenNoExplicitCompleterTargetSpecified_test(): + app = TestApp( handlers.app ) + subcommands_data = BuildRequest( filetype = 'python' ) + + eq_( [ 'GoToDefinition', + 'GoToDeclaration', + 'GoToDefinitionElseDeclaration' ], + app.post_json( '/defined_subcommands', subcommands_data ).json ) + + +@with_setup( Setup ) +def Diagnostics_ClangCompleter_ZeroBasedLineAndColumn_test(): + app = TestApp( handlers.app ) + contents = """ +struct Foo { + int x // semicolon missing here! + int y; + int c; + int d; +}; +""" + + event_data = BuildRequest( compilation_flags = ['-x', 'c++'], + event_name = 'FileReadyToParse', + contents = contents, + filetype = 'cpp' ) + + results = app.post_json( '/event_notification', event_data ).json + assert_that( results, + contains( + has_entries( { 'text': contains_string( "expected ';'" ), + 'line_num': 2, + 'column_num': 7 } ) ) ) + + +@with_setup( Setup ) +def GetDetailedDiagnostic_ClangCompleter_Works_test(): + app = TestApp( handlers.app ) + contents = """ +struct Foo { + int x // semicolon missing here! + int y; + int c; + int d; +}; +""" + + diag_data = BuildRequest( compilation_flags = ['-x', 'c++'], + line_num = 2, + contents = contents, + filetype = 'cpp' ) + + event_data = diag_data.copy() + event_data.update( { + 'event_name': 'FileReadyToParse', + } ) + + app.post_json( '/event_notification', event_data ) + results = app.post_json( '/detailed_diagnostic', diag_data ).json + assert_that( results, + has_entry( 'message', contains_string( "expected ';'" ) ) ) + + +@with_setup( Setup ) +def FiletypeCompletionAvailable_Works_test(): + app = TestApp( handlers.app ) + request_data = { + 'filetypes': ['python'] + } + + ok_( app.post_json( '/semantic_completion_available', + request_data ).json ) + + +@with_setup( Setup ) +def UserOptions_Works_test(): + app = TestApp( handlers.app ) + options = app.get( '/user_options' ).json + ok_( len( options ) ) + + options[ 'foobar' ] = 'zoo' + + app.post_json( '/user_options', options ) + eq_( options, app.get( '/user_options' ).json ) + diff --git a/python/ycm/server/tests/testdata/.ycm_extra_conf.py b/python/ycm/server/tests/testdata/.ycm_extra_conf.py new file mode 100644 index 00000000..b94aed95 --- /dev/null +++ b/python/ycm/server/tests/testdata/.ycm_extra_conf.py @@ -0,0 +1,5 @@ +def FlagsForFile( filename ): + return { + 'flags': ['-x', 'c++'], + 'do_cache': True + } diff --git a/python/ycm/server/tests/testdata/basic.cpp b/python/ycm/server/tests/testdata/basic.cpp new file mode 100644 index 00000000..a20e2dd0 --- /dev/null +++ b/python/ycm/server/tests/testdata/basic.cpp @@ -0,0 +1,13 @@ +struct Foo { + int x; + int y; + char c; +}; + +int main() +{ + Foo foo; + // The location after the dot is line 11, col 7 + foo. +} + diff --git a/python/ycm/server/tests/testdata/testy/Program.cs b/python/ycm/server/tests/testdata/testy/Program.cs new file mode 100644 index 00000000..cac606c8 --- /dev/null +++ b/python/ycm/server/tests/testdata/testy/Program.cs @@ -0,0 +1,12 @@ +using System; + +namespace testy +{ + class MainClass + { + public static void Main (string[] args) + { + Console. + } + } +} diff --git a/python/ycm/server/tests/testdata/testy/Properties/AssemblyInfo.cs b/python/ycm/server/tests/testdata/testy/Properties/AssemblyInfo.cs new file mode 100644 index 00000000..bed715b8 --- /dev/null +++ b/python/ycm/server/tests/testdata/testy/Properties/AssemblyInfo.cs @@ -0,0 +1,22 @@ +using System.Reflection; +using System.Runtime.CompilerServices; + +// Information about this assembly is defined by the following attributes. +// Change them to the values specific to your project. +[assembly: AssemblyTitle ("testy")] +[assembly: AssemblyDescription ("")] +[assembly: AssemblyConfiguration ("")] +[assembly: AssemblyCompany ("")] +[assembly: AssemblyProduct ("")] +[assembly: AssemblyCopyright ("valloric")] +[assembly: AssemblyTrademark ("")] +[assembly: AssemblyCulture ("")] +// The assembly version has the format "{Major}.{Minor}.{Build}.{Revision}". +// The form "{Major}.{Minor}.*" will automatically update the build and revision, +// and "{Major}.{Minor}.{Build}.*" will update just the revision. +[assembly: AssemblyVersion ("1.0.*")] +// The following attributes are used to specify the signing key for the assembly, +// if desired. See the Mono documentation for more information about signing. +//[assembly: AssemblyDelaySign(false)] +//[assembly: AssemblyKeyFile("")] + diff --git a/python/ycm/server/tests/testdata/testy/testy.csproj b/python/ycm/server/tests/testdata/testy/testy.csproj new file mode 100644 index 00000000..1f1c0dd1 --- /dev/null +++ b/python/ycm/server/tests/testdata/testy/testy.csproj @@ -0,0 +1,41 @@ + + + + Debug + x86 + 10.0.0 + 2.0 + {0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F} + Exe + testy + testy + + + true + full + false + bin\Debug + DEBUG; + prompt + 4 + true + x86 + + + full + true + bin\Release + prompt + 4 + true + x86 + + + + + + + + + + \ No newline at end of file diff --git a/python/ycm/server/tests/testdata/testy/testy.sln b/python/ycm/server/tests/testdata/testy/testy.sln new file mode 100644 index 00000000..8c266824 --- /dev/null +++ b/python/ycm/server/tests/testdata/testy/testy.sln @@ -0,0 +1,20 @@ + +Microsoft Visual Studio Solution File, Format Version 11.00 +# Visual Studio 2010 +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "testy", "testy.csproj", "{0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|x86 = Debug|x86 + Release|x86 = Release|x86 + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}.Debug|x86.ActiveCfg = Debug|x86 + {0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}.Debug|x86.Build.0 = Debug|x86 + {0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}.Release|x86.ActiveCfg = Release|x86 + {0C99F719-E00E-4CCD-AB9F-FEFBCD97C51F}.Release|x86.Build.0 = Release|x86 + EndGlobalSection + GlobalSection(MonoDevelopProperties) = preSolution + StartupItem = testy.csproj + EndGlobalSection +EndGlobal diff --git a/python/ycm/server/tests/testdata/testy/testy.userprefs b/python/ycm/server/tests/testdata/testy/testy.userprefs new file mode 100644 index 00000000..1895a66a --- /dev/null +++ b/python/ycm/server/tests/testdata/testy/testy.userprefs @@ -0,0 +1,13 @@ + + + + + + + + + + + + + \ No newline at end of file diff --git a/python/ycm/server/watchdog_plugin.py b/python/ycm/server/watchdog_plugin.py new file mode 100644 index 00000000..23c48ae6 --- /dev/null +++ b/python/ycm/server/watchdog_plugin.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +import time +import os +import copy +from ycm import utils +from threading import Thread, Lock + +# This class implements the Bottle plugin API: +# http://bottlepy.org/docs/dev/plugindev.html +# +# The idea here is to decorate every route handler automatically so that on +# every request, we log when the request was made. Then a watchdog thread checks +# every check_interval_seconds whether the server has been idle for a time +# greater that the passed-in idle_suicide_seconds. If it has, we kill the +# server. +# +# We want to do this so that if something goes bonkers in Vim and the server +# never gets killed by the client, we don't end up with lots of zombie servers. +class WatchdogPlugin( object ): + name = 'watchdog' + api = 2 + + + def __init__( self, + idle_suicide_seconds, + check_interval_seconds = 60 * 10 ): + self._check_interval_seconds = check_interval_seconds + self._idle_suicide_seconds = idle_suicide_seconds + self._last_request_time = time.time() + self._last_request_time_lock = Lock() + if idle_suicide_seconds <= 0: + return + self._watchdog_thread = Thread( target = self._WatchdogMain ) + self._watchdog_thread.daemon = True + self._watchdog_thread.start() + + + def _GetLastRequestTime( self ): + with self._last_request_time_lock: + return copy.deepcopy( self._last_request_time ) + + + def _SetLastRequestTime( self, new_value ): + with self._last_request_time_lock: + self._last_request_time = new_value + + + def _WatchdogMain( self ): + while True: + time.sleep( self._check_interval_seconds ) + if time.time() - self._GetLastRequestTime() > self._idle_suicide_seconds: + utils.TerminateProcess( os.getpid() ) + + + def __call__( self, callback ): + def wrapper( *args, **kwargs ): + self._SetLastRequestTime( time.time() ) + return callback( *args, **kwargs ) + return wrapper + diff --git a/python/ycm/server/ycmd.py b/python/ycm/server/ycmd.py new file mode 100755 index 00000000..07d34d68 --- /dev/null +++ b/python/ycm/server/ycmd.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +from server_utils import SetUpPythonPath +SetUpPythonPath() + +import sys +import logging +import json +import argparse +import waitress +import signal +from ycm import user_options_store +from ycm import extra_conf_store +from ycm.server.watchdog_plugin import WatchdogPlugin + + +def YcmCoreSanityCheck(): + if 'ycm_core' in sys.modules: + raise RuntimeError( 'ycm_core already imported, ycmd has a bug!' ) + + +# We manually call sys.exit() on SIGTERM and SIGINT so that atexit handlers are +# properly executed. +def SetUpSignalHandler(): + def SignalHandler( signum, frame ): + sys.exit() + + for sig in [ signal.SIGTERM, + signal.SIGINT ]: + signal.signal( sig, SignalHandler ) + + +def Main(): + parser = argparse.ArgumentParser() + parser.add_argument( '--host', type = str, default = 'localhost', + help = 'server hostname') + parser.add_argument( '--port', type = int, default = 6666, + help = 'server port') + parser.add_argument( '--log', type = str, default = 'info', + help = 'log level, one of ' + '[debug|info|warning|error|critical]' ) + parser.add_argument( '--idle_suicide_seconds', type = int, default = 0, + help = 'num idle seconds before server shuts down') + parser.add_argument( '--options_file', type = str, default = '', + help = 'file with user options, in JSON format' ) + args = parser.parse_args() + + numeric_level = getattr( logging, args.log.upper(), None ) + if not isinstance( numeric_level, int ): + raise ValueError( 'Invalid log level: %s' % args.log ) + + # Has to be called before any call to logging.getLogger() + logging.basicConfig( format = '%(asctime)s - %(levelname)s - %(message)s', + level = numeric_level ) + + options = None + if args.options_file: + options = json.load( open( args.options_file, 'r' ) ) + user_options_store.SetAll( options ) + # This ensures that ycm_core is not loaded before extra conf + # preload was run. + YcmCoreSanityCheck() + extra_conf_store.CallGlobalExtraConfYcmCorePreloadIfExists() + + # This can't be a top-level import because it transitively imports + # ycm_core which we want to be imported ONLY after extra conf + # preload has executed. + from ycm.server import handlers + handlers.UpdateUserOptions( options ) + SetUpSignalHandler() + handlers.app.install( WatchdogPlugin( args.idle_suicide_seconds ) ) + waitress.serve( handlers.app, + host = args.host, + port = args.port, + threads = 10 ) + + +if __name__ == "__main__": + Main() + diff --git a/python/ycm/user_options_store.py b/python/ycm/user_options_store.py new file mode 100644 index 00000000..fdb422a9 --- /dev/null +++ b/python/ycm/user_options_store.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python +# +# Copyright (C) 2013 Strahinja Val Markovic +# +# This file is part of YouCompleteMe. +# +# YouCompleteMe is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# YouCompleteMe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with YouCompleteMe. If not, see . + +import json +import os +from frozendict import frozendict + +_USER_OPTIONS = {} + +def SetAll( new_options ): + global _USER_OPTIONS + _USER_OPTIONS = frozendict( new_options ) + + +def GetAll(): + return _USER_OPTIONS + + +def Value( key ): + return _USER_OPTIONS[ key ] + + +def LoadDefaults(): + SetAll( DefaultOptions() ) + + +def DefaultOptions(): + settings_path = os.path.join( + os.path.dirname( os.path.abspath( __file__ ) ), + 'server/default_settings.json' ) + with open( settings_path ) as f: + return json.loads( f.read() ) + diff --git a/python/ycm/utils.py b/python/ycm/utils.py index d1fe5035..1e63f4f3 100644 --- a/python/ycm/utils.py +++ b/python/ycm/utils.py @@ -17,6 +17,18 @@ # You should have received a copy of the GNU General Public License # along with YouCompleteMe. If not, see . +import tempfile +import os +import sys +import signal +import functools +import socket +import stat + +WIN_PYTHON27_PATH = 'C:\python27\pythonw.exe' +WIN_PYTHON26_PATH = 'C:\python26\pythonw.exe' + + def IsIdentifierChar( char ): return char.isalnum() or char == '_' @@ -24,3 +36,99 @@ def IsIdentifierChar( char ): def SanitizeQuery( query ): return query.strip() + +def ToUtf8IfNeeded( string_or_unicode ): + if isinstance( string_or_unicode, unicode ): + return string_or_unicode.encode( 'utf8' ) + return string_or_unicode + + +def PathToTempDir(): + tempdir = os.path.join( tempfile.gettempdir(), 'ycm_temp' ) + if not os.path.exists( tempdir ): + os.makedirs( tempdir ) + # Needed to support multiple users working on the same machine; + # see issue 606. + MakeFolderAccessibleToAll( tempdir ) + return tempdir + + +def MakeFolderAccessibleToAll( path_to_folder ): + current_stat = os.stat( path_to_folder ) + # readable, writable and executable by everyone + flags = ( current_stat.st_mode | stat.S_IROTH | stat.S_IWOTH | stat.S_IXOTH + | stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP ) + os.chmod( path_to_folder, flags ) + + +def GetUnusedLocalhostPort(): + sock = socket.socket() + # This tells the OS to give us any free port in the range [1024 - 65535] + sock.bind( ( '', 0 ) ) + port = sock.getsockname()[ 1 ] + sock.close() + return port + + +def PathToPythonInterpreter(): + # This is a bit tricky. Normally, sys.executable has the full path to the + # Python interpreter. But this code is also executed from inside Vim's + # embedded Python. On Unix machines, even that Python returns a good value for + # sys.executable, but on Windows it returns the path to the Vim binary, which + # is useless to us (issue #581). So we check the common install location for + # Python on Windows, first for Python 2.7 and then for 2.6. + # + # I'm open to better ideas on how to do this. + + if OnWindows(): + if os.path.exists( WIN_PYTHON27_PATH ): + return WIN_PYTHON27_PATH + elif os.path.exists( WIN_PYTHON26_PATH ): + return WIN_PYTHON26_PATH + raise RuntimeError( 'Python 2.7/2.6 not installed!' ) + else: + return sys.executable + + +def OnWindows(): + return sys.platform == 'win32' + + +# From here: http://stackoverflow.com/a/8536476/1672783 +def TerminateProcess( pid ): + if OnWindows(): + import ctypes + PROCESS_TERMINATE = 1 + handle = ctypes.windll.kernel32.OpenProcess( PROCESS_TERMINATE, + False, + pid ) + ctypes.windll.kernel32.TerminateProcess( handle, -1 ) + ctypes.windll.kernel32.CloseHandle( handle ) + else: + os.kill( pid, signal.SIGTERM ) + + +def AddThirdPartyFoldersToSysPath(): + path_to_third_party = os.path.join( + os.path.dirname( os.path.abspath( __file__ ) ), + '../../third_party' ) + + for folder in os.listdir( path_to_third_party ): + sys.path.insert( 0, os.path.realpath( os.path.join( path_to_third_party, + folder ) ) ) + +def Memoize( obj ): + cache = obj.cache = {} + + @functools.wraps( obj ) + def memoizer( *args, **kwargs ): + key = str( args ) + str( kwargs ) + if key not in cache: + cache[ key ] = obj( *args, **kwargs ) + return cache[ key ] + return memoizer + + +def ForceSemanticCompletion( request_data ): + return ( 'force_semantic' in request_data and + bool( request_data[ 'force_semantic' ] ) ) diff --git a/python/ycm/vimsupport.py b/python/ycm/vimsupport.py index bd13041d..6ee36e3a 100644 --- a/python/ycm/vimsupport.py +++ b/python/ycm/vimsupport.py @@ -18,6 +18,8 @@ # along with YouCompleteMe. If not, see . import vim +import os +import json def CurrentLineAndColumn(): """Returns the 0-based current line and 0-based current column.""" @@ -46,12 +48,74 @@ def TextAfterCursor(): return vim.current.line[ CurrentColumn(): ] -def GetUnsavedBuffers(): - def BufferModified( buffer_number ): - to_eval = 'getbufvar({0}, "&mod")'.format( buffer_number ) - return GetBoolValue( to_eval ) +# Note the difference between buffer OPTIONS and VARIABLES; the two are not +# the same. +def GetBufferOption( buffer_object, option ): + # The 'options' property is only available in recent (7.4+) Vim builds + if hasattr( buffer_object, 'options' ): + return buffer_object.options[ option ] - return ( x for x in vim.buffers if BufferModified( x.number ) ) + to_eval = 'getbufvar({0}, "&{1}")'.format( buffer_object.number, option ) + return GetVariableValue( to_eval ) + + +def GetUnsavedAndCurrentBufferData(): + def BufferModified( buffer_object ): + return bool( int( GetBufferOption( buffer_object, 'mod' ) ) ) + + buffers_data = {} + for buffer_object in vim.buffers: + if not ( BufferModified( buffer_object ) or + buffer_object == vim.current.buffer ): + continue + + buffers_data[ GetBufferFilepath( buffer_object ) ] = { + 'contents': '\n'.join( buffer_object ), + 'filetypes': FiletypesForBuffer( buffer_object ) + } + + return buffers_data + + +def GetBufferNumberForFilename( filename, open_file_if_needed = True ): + return int( vim.eval( "bufnr('{0}', {1})".format( + os.path.realpath( filename ), + int( open_file_if_needed ) ) ) ) + + +def GetCurrentBufferFilepath(): + return GetBufferFilepath( vim.current.buffer ) + + +def GetBufferFilepath( buffer_object ): + if buffer_object.name: + return buffer_object.name + # Buffers that have just been created by a command like :enew don't have any + # buffer name so we use the buffer number for that. + return os.path.join( os.getcwd(), str( buffer_object.number ) ) + + +# Given a dict like {'a': 1}, loads it into Vim as if you ran 'let g:a = 1' +# When |overwrite| is True, overwrites the existing value in Vim. +def LoadDictIntoVimGlobals( new_globals, overwrite = True ): + extend_option = '"force"' if overwrite else '"keep"' + + # We need to use json.dumps because that won't use the 'u' prefix on strings + # which Vim would bork on. + vim.eval( 'extend( g:, {0}, {1})'.format( json.dumps( new_globals ), + extend_option ) ) + + +# Changing the returned dict will NOT change the value in Vim. +def GetReadOnlyVimGlobals( force_python_objects = False ): + if force_python_objects: + return vim.eval( 'g:' ) + + try: + # vim.vars is fairly new so it might not exist + return vim.vars + except: + return vim.eval( 'g:' ) # Both |line| and |column| need to be 1-based @@ -59,7 +123,7 @@ def JumpToLocation( filename, line, column ): # Add an entry to the jumplist vim.command( "normal! m'" ) - if filename != vim.current.buffer.name: + if filename != GetCurrentBufferFilepath(): # We prefix the command with 'keepjumps' so that opening the file is not # recorded in the jumplist. So when we open the file and move the cursor to # a location in it, the user can use CTRL-O to jump back to the original @@ -73,18 +137,24 @@ def JumpToLocation( filename, line, column ): vim.command( 'normal! zz' ) -def NumLinesInBuffer( buffer ): +def NumLinesInBuffer( buffer_object ): # This is actually less than obvious, that's why it's wrapped in a function - return len( buffer ) + return len( buffer_object ) +# Calling this function from the non-GUI thread will sometimes crash Vim. At the +# time of writing, YCM only uses the GUI thread inside Vim (this used to not be +# the case). def PostVimMessage( message ): - # TODO: Check are we on the main thread or not, and if not, force a crash - # here. This should make it impossible to accidentally call this from a - # non-GUI thread which *sometimes* crashes Vim because Vim is not thread-safe. - # A consistent crash should force us to notice the error. - vim.command( "echohl WarningMsg | echomsg '{0}' | echohl None" - .format( EscapeForVim( message ) ) ) + vim.command( "echohl WarningMsg | echom '{0}' | echohl None" + .format( EscapeForVim( str( message ) ) ) ) + +# Unlike PostVimMesasge, this supports messages with newlines in them because it +# uses 'echo' instead of 'echomsg'. This also means that the message will NOT +# appear in Vim's message log. +def PostMultiLineNotice( message ): + vim.command( "echohl WarningMsg | echo '{0}' | echohl None" + .format( EscapeForVim( str( message ) ) ) ) def PresentDialog( message, choices, default_choice_index = 0 ): @@ -128,15 +198,13 @@ def EscapeForVim( text ): def CurrentFiletypes(): - ft_string = vim.eval( "&filetype" ) - return ft_string.split( '.' ) + return vim.eval( "&filetype" ).split( '.' ) def FiletypesForBuffer( buffer_object ): # NOTE: Getting &ft for other buffers only works when the buffer has been # visited by the user at least once, which is true for modified buffers - ft_string = vim.eval( 'getbufvar({0}, "&ft")'.format( buffer_object.number ) ) - return ft_string.split( '.' ) + return GetBufferOption( buffer_object, 'ft' ).split( '.' ) def GetVariableValue( variable ): @@ -145,3 +213,8 @@ def GetVariableValue( variable ): def GetBoolValue( variable ): return bool( int( vim.eval( variable ) ) ) + + +def GetIntValue( variable ): + return int( vim.eval( variable ) ) + diff --git a/python/ycm/youcompleteme.py b/python/ycm/youcompleteme.py index ee689168..bf9d4d99 100644 --- a/python/ycm/youcompleteme.py +++ b/python/ycm/youcompleteme.py @@ -17,203 +17,301 @@ # You should have received a copy of the GNU General Public License # along with YouCompleteMe. If not, see . -import imp import os import vim -import ycm_core +import subprocess +import tempfile +import json from ycm import vimsupport +from ycm import utils from ycm.completers.all.omni_completer import OmniCompleter -from ycm.completers.general.general_completer_store import GeneralCompleterStore +from ycm.completers.general import syntax_parse +from ycm.completers.completer_utils import FiletypeCompleterExistsForFiletype +from ycm.client.base_request import BaseRequest, BuildRequestData +from ycm.client.command_request import SendCommandRequest +from ycm.client.completion_request import CompletionRequest +from ycm.client.omni_completion_request import OmniCompletionRequest +from ycm.client.event_notification import ( SendEventNotificationAsync, + EventNotification ) +from ycm.server.responses import ServerError +try: + from UltiSnips import UltiSnips_Manager + USE_ULTISNIPS_DATA = True +except ImportError: + USE_ULTISNIPS_DATA = False -FILETYPE_SPECIFIC_COMPLETION_TO_DISABLE = vim.eval( - 'g:ycm_filetype_specific_completion_to_disable' ) +SERVER_CRASH_MESSAGE_STDERR_FILE = 'The ycmd server SHUT DOWN with output:\n' +SERVER_CRASH_MESSAGE_SAME_STDERR = ( + 'The ycmd server shut down, check console output for logs!' ) class YouCompleteMe( object ): - def __init__( self ): - self.gencomp = GeneralCompleterStore() - self.omnicomp = OmniCompleter() - self.filetype_completers = {} + def __init__( self, user_options ): + self._user_options = user_options + self._omnicomp = OmniCompleter( user_options ) + self._latest_completion_request = None + self._latest_file_parse_request = None + self._server_stdout = None + self._server_stderr = None + self._server_popen = None + self._filetypes_with_keywords_loaded = set() + self._temp_options_filename = None + self._SetupServer() - def GetGeneralCompleter( self ): - return self.gencomp + def _SetupServer( self ): + server_port = utils.GetUnusedLocalhostPort() + with tempfile.NamedTemporaryFile( delete = False ) as options_file: + self._temp_options_filename = options_file.name + json.dump( dict( self._user_options ), options_file ) + args = [ utils.PathToPythonInterpreter(), + _PathToServerScript(), + '--port={0}'.format( server_port ), + '--options_file={0}'.format( options_file.name ), + '--log={0}'.format( self._user_options[ 'server_log_level' ] ), + '--idle_suicide_seconds={0}'.format( + self._user_options[ 'server_idle_suicide_seconds' ] ) ] + + BaseRequest.server_location = 'http://localhost:' + str( server_port ) + + if self._user_options[ 'server_use_vim_stdout' ]: + self._server_popen = subprocess.Popen( args ) + else: + filename_format = os.path.join( utils.PathToTempDir(), + 'server_{port}_{std}.log' ) + + self._server_stdout = filename_format.format( port = server_port, + std = 'stdout' ) + self._server_stderr = filename_format.format( port = server_port, + std = 'stderr' ) + + with open( self._server_stderr, 'w' ) as fstderr: + with open( self._server_stdout, 'w' ) as fstdout: + self._server_popen = subprocess.Popen( args, + stdout = fstdout, + stderr = fstderr ) + self._NotifyUserIfServerCrashed() + + + def _IsServerAlive( self ): + returncode = self._server_popen.poll() + # When the process hasn't finished yet, poll() returns None. + return returncode is None + + + def _NotifyUserIfServerCrashed( self ): + if self._IsServerAlive(): + return + if self._server_stderr: + with open( self._server_stderr, 'r' ) as server_stderr_file: + vimsupport.PostMultiLineNotice( SERVER_CRASH_MESSAGE_STDERR_FILE + + server_stderr_file.read() ) + else: + vimsupport.PostVimMessage( SERVER_CRASH_MESSAGE_SAME_STDERR ) + + + def ServerPid( self ): + if not self._server_popen: + return -1 + return self._server_popen.pid + + + def RestartServer( self ): + vimsupport.PostVimMessage( 'Restarting ycmd server...' ) + self.OnVimLeave() + self._SetupServer() + + + def CreateCompletionRequest( self, force_semantic = False ): + # We have to store a reference to the newly created CompletionRequest + # because VimScript can't store a reference to a Python object across + # function calls... Thus we need to keep this request somewhere. + if ( not self.NativeFiletypeCompletionAvailable() and + self.CurrentFiletypeCompletionEnabled() and + self._omnicomp.ShouldUseNow() ): + self._latest_completion_request = OmniCompletionRequest( self._omnicomp ) + else: + self._latest_completion_request = ( CompletionRequest( force_semantic ) + if self._IsServerAlive() else + None ) + return self._latest_completion_request + + + def SendCommandRequest( self, arguments, completer ): + if self._IsServerAlive(): + return SendCommandRequest( arguments, completer ) + + + def GetDefinedSubcommands( self ): + if self._IsServerAlive(): + return BaseRequest.PostDataToHandler( BuildRequestData(), + 'defined_subcommands' ) + else: + return [] + + + def GetCurrentCompletionRequest( self ): + return self._latest_completion_request def GetOmniCompleter( self ): - return self.omnicomp - - - def GetFiletypeCompleter( self ): - filetypes = vimsupport.CurrentFiletypes() - - completers = [ self.GetFiletypeCompleterForFiletype( filetype ) - for filetype in filetypes ] - - if not completers: - return None - - # Try to find a native completer first - for completer in completers: - if completer and completer is not self.omnicomp: - return completer - - # Return the omni completer for the first filetype - return completers[0] - - - def GetFiletypeCompleterForFiletype( self, filetype ): - try: - return self.filetype_completers[ filetype ] - except KeyError: - pass - - module_path = _PathToFiletypeCompleterPluginLoader( filetype ) - - completer = None - supported_filetypes = [ filetype ] - if os.path.exists( module_path ): - module = imp.load_source( filetype, module_path ) - completer = module.GetCompleter() - if completer: - supported_filetypes.extend( completer.SupportedFiletypes() ) - else: - completer = self.omnicomp - - for supported_filetype in supported_filetypes: - self.filetype_completers[ supported_filetype ] = completer - return completer - - - def ShouldUseGeneralCompleter( self, start_column ): - return self.gencomp.ShouldUseNow( start_column ) - - - def ShouldUseFiletypeCompleter( self, start_column ): - if self.FiletypeCompletionUsable(): - return self.GetFiletypeCompleter().ShouldUseNow( - start_column ) - return False + return self._omnicomp def NativeFiletypeCompletionAvailable( self ): - completer = self.GetFiletypeCompleter() - return bool( completer ) and completer is not self.omnicomp - - - def FiletypeCompletionAvailable( self ): - return bool( self.GetFiletypeCompleter() ) + return any( [ FiletypeCompleterExistsForFiletype( x ) for x in + vimsupport.CurrentFiletypes() ] ) def NativeFiletypeCompletionUsable( self ): - return ( _CurrentFiletypeCompletionEnabled() and + return ( self.CurrentFiletypeCompletionEnabled() and self.NativeFiletypeCompletionAvailable() ) - def FiletypeCompletionUsable( self ): - return ( _CurrentFiletypeCompletionEnabled() and - self.FiletypeCompletionAvailable() ) - - def OnFileReadyToParse( self ): - self.gencomp.OnFileReadyToParse() + self._omnicomp.OnFileReadyToParse( None ) - if self.FiletypeCompletionUsable(): - self.GetFiletypeCompleter().OnFileReadyToParse() + if not self._IsServerAlive(): + self._NotifyUserIfServerCrashed() + + extra_data = {} + if self._user_options[ 'collect_identifiers_from_tags_files' ]: + extra_data[ 'tag_files' ] = _GetTagFiles() + + if self._user_options[ 'seed_identifiers_with_syntax' ]: + self._AddSyntaxDataIfNeeded( extra_data ) + + self._latest_file_parse_request = EventNotification( 'FileReadyToParse', + extra_data ) + self._latest_file_parse_request.Start() def OnBufferUnload( self, deleted_buffer_file ): - self.gencomp.OnBufferUnload( deleted_buffer_file ) - - if self.FiletypeCompletionUsable(): - self.GetFiletypeCompleter().OnBufferUnload( deleted_buffer_file ) + if not self._IsServerAlive(): + return + SendEventNotificationAsync( 'BufferUnload', + { 'unloaded_buffer': deleted_buffer_file } ) def OnBufferVisit( self ): - self.gencomp.OnBufferVisit() - - if self.FiletypeCompletionUsable(): - self.GetFiletypeCompleter().OnBufferVisit() + if not self._IsServerAlive(): + return + extra_data = {} + _AddUltiSnipsDataIfNeeded( extra_data ) + SendEventNotificationAsync( 'BufferVisit', extra_data ) def OnInsertLeave( self ): - self.gencomp.OnInsertLeave() - - if self.FiletypeCompletionUsable(): - self.GetFiletypeCompleter().OnInsertLeave() + if not self._IsServerAlive(): + return + SendEventNotificationAsync( 'InsertLeave' ) def OnVimLeave( self ): - self.gencomp.OnVimLeave() + if self._IsServerAlive(): + self._server_popen.terminate() + os.remove( self._temp_options_filename ) - if self.FiletypeCompletionUsable(): - self.GetFiletypeCompleter().OnVimLeave() + if not self._user_options[ 'server_keep_logfiles' ]: + if self._server_stderr: + os.remove( self._server_stderr ) + if self._server_stdout: + os.remove( self._server_stdout ) + + + def OnCurrentIdentifierFinished( self ): + if not self._IsServerAlive(): + return + SendEventNotificationAsync( 'CurrentIdentifierFinished' ) def DiagnosticsForCurrentFileReady( self ): - if self.FiletypeCompletionUsable(): - return self.GetFiletypeCompleter().DiagnosticsForCurrentFileReady() - return False + return bool( self._latest_file_parse_request and + self._latest_file_parse_request.Done() ) - def GetDiagnosticsForCurrentFile( self ): - if self.FiletypeCompletionUsable(): - return self.GetFiletypeCompleter().GetDiagnosticsForCurrentFile() + def GetDiagnosticsFromStoredRequest( self ): + if self.DiagnosticsForCurrentFileReady(): + to_return = self._latest_file_parse_request.Response() + # We set the diagnostics request to None because we want to prevent + # Syntastic from repeatedly refreshing the buffer with the same diags. + # Setting this to None makes DiagnosticsForCurrentFileReady return False + # until the next request is created. + self._latest_file_parse_request = None + return to_return return [] def ShowDetailedDiagnostic( self ): - if self.FiletypeCompletionUsable(): - return self.GetFiletypeCompleter().ShowDetailedDiagnostic() - - - def GettingCompletions( self ): - if self.FiletypeCompletionUsable(): - return self.GetFiletypeCompleter().GettingCompletions() - return False - - - def OnCurrentIdentifierFinished( self ): - self.gencomp.OnCurrentIdentifierFinished() - - if self.FiletypeCompletionUsable(): - self.GetFiletypeCompleter().OnCurrentIdentifierFinished() + if not self._IsServerAlive(): + return + try: + debug_info = BaseRequest.PostDataToHandler( BuildRequestData(), + 'detailed_diagnostic' ) + if 'message' in debug_info: + vimsupport.EchoText( debug_info[ 'message' ] ) + except ServerError as e: + vimsupport.PostVimMessage( str( e ) ) def DebugInfo( self ): - completers = set( self.filetype_completers.values() ) - completers.add( self.gencomp ) - output = [] - for completer in completers: - if not completer: - continue - debug = completer.DebugInfo() - if debug: - output.append( debug ) + if self._IsServerAlive(): + debug_info = BaseRequest.PostDataToHandler( BuildRequestData(), + 'debug_info' ) + else: + debug_info = 'Server crashed, no debug info from server' + debug_info += '\nServer running at: {0}'.format( + BaseRequest.server_location ) + debug_info += '\nServer process ID: {0}'.format( self._server_popen.pid ) + if self._server_stderr or self._server_stdout: + debug_info += '\nServer logfiles:\n {0}\n {1}'.format( + self._server_stdout, + self._server_stderr ) - has_clang_support = ycm_core.HasClangSupport() - output.append( 'Has Clang support compiled in: {0}'.format( - has_clang_support ) ) - - if has_clang_support: - output.append( ycm_core.ClangVersion() ) - - return '\n'.join( output ) + return debug_info -def _CurrentFiletypeCompletionEnabled(): - filetypes = vimsupport.CurrentFiletypes() - return not all([ x in FILETYPE_SPECIFIC_COMPLETION_TO_DISABLE - for x in filetypes ]) + def CurrentFiletypeCompletionEnabled( self ): + filetypes = vimsupport.CurrentFiletypes() + filetype_to_disable = self._user_options[ + 'filetype_specific_completion_to_disable' ] + return not all([ x in filetype_to_disable for x in filetypes ]) -def _PathToCompletersFolder(): + def _AddSyntaxDataIfNeeded( self, extra_data ): + filetype = vimsupport.CurrentFiletypes()[ 0 ] + if filetype in self._filetypes_with_keywords_loaded: + return + + self._filetypes_with_keywords_loaded.add( filetype ) + extra_data[ 'syntax_keywords' ] = list( + syntax_parse.SyntaxKeywordsForCurrentBuffer() ) + + +def _GetTagFiles(): + tag_files = vim.eval( 'tagfiles()' ) + current_working_directory = os.getcwd() + return [ os.path.join( current_working_directory, x ) for x in tag_files ] + + +def _PathToServerScript(): dir_of_current_script = os.path.dirname( os.path.abspath( __file__ ) ) - return os.path.join( dir_of_current_script, 'completers' ) + return os.path.join( dir_of_current_script, 'server/ycmd.py' ) -def _PathToFiletypeCompleterPluginLoader( filetype ): - return os.path.join( _PathToCompletersFolder(), filetype, 'hook.py' ) +def _AddUltiSnipsDataIfNeeded( extra_data ): + if not USE_ULTISNIPS_DATA: + return + try: + rawsnips = UltiSnips_Manager._snips( '', 1 ) + except: + return + # UltiSnips_Manager._snips() returns a class instance where: + # class.trigger - name of snippet trigger word ( e.g. defn or testcase ) + # class.description - description of the snippet + extra_data[ 'ultisnips_snippets' ] = [ { 'trigger': x.trigger, + 'description': x.description + } for x in rawsnips ] diff --git a/run_tests.sh b/run_tests.sh new file mode 100755 index 00000000..06e00e46 --- /dev/null +++ b/run_tests.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash + +set -e + +function usage { + echo "Usage: $0 [--no-clang-completer]" + exit 0 +} + +flake8 --select=F,C9 --max-complexity=10 python + +use_clang_completer=true +for flag in $@; do + case "$flag" in + --no-clang-completer) + use_clang_completer=false + ;; + *) + usage + ;; + esac +done + +if [ -n "$USE_CLANG_COMPLETER" ]; then + use_clang_completer=$USE_CLANG_COMPLETER +fi + +if $use_clang_completer; then + extra_cmake_args="-DUSE_CLANG_COMPLETER=ON -DUSE_DEV_FLAGS=ON" +else + extra_cmake_args="-DUSE_DEV_FLAGS=ON" +fi + +EXTRA_CMAKE_ARGS=$extra_cmake_args YCM_TESTRUN=1 ./install.sh --omnisharp-completer + +for directory in third_party/*; do + if [ -d "${directory}" ]; then + export PYTHONPATH=$PWD/${directory}:$PYTHONPATH + fi +done + +if $use_clang_completer; then + nosetests -v python +else + nosetests -v --exclude=".*Clang.*" python +fi diff --git a/style_format.sh b/style_format.sh index 2988c68b..d0983cf5 100755 --- a/style_format.sh +++ b/style_format.sh @@ -19,6 +19,7 @@ astyle \ --recursive \ --exclude=gmock \ --exclude=testdata \ +--exclude=ycm_client_support.cpp \ --exclude=ycm_core.cpp \ --exclude=CustomAssert.h \ --exclude=CustomAssert.cpp \ diff --git a/third_party/argparse b/third_party/argparse new file mode 160000 index 00000000..46af816d --- /dev/null +++ b/third_party/argparse @@ -0,0 +1 @@ +Subproject commit 46af816db4812eab5f4639717bf1ad2eb17cc1ff diff --git a/third_party/bottle b/third_party/bottle new file mode 160000 index 00000000..154369b2 --- /dev/null +++ b/third_party/bottle @@ -0,0 +1 @@ +Subproject commit 154369b2b9f7393ca9d3d73de7e046e2342cf00a diff --git a/third_party/frozendict b/third_party/frozendict new file mode 160000 index 00000000..2ea45f3f --- /dev/null +++ b/third_party/frozendict @@ -0,0 +1 @@ +Subproject commit 2ea45f3f429c5283f9f24738812f0ee152a8f4c1 diff --git a/third_party/jedi b/third_party/jedi new file mode 160000 index 00000000..099fe4ee --- /dev/null +++ b/third_party/jedi @@ -0,0 +1 @@ +Subproject commit 099fe4eeb3544005a8e2ffdaae43fd8a12c82f16 diff --git a/third_party/pythonfutures/CHANGES b/third_party/pythonfutures/CHANGES new file mode 100755 index 00000000..81df636f --- /dev/null +++ b/third_party/pythonfutures/CHANGES @@ -0,0 +1,44 @@ +2.1.4 +===== + +- Ported the library again from Python 3.2.5 to get the latest bug fixes + + +2.1.3 +===== + +- Fixed race condition in wait(return_when=ALL_COMPLETED) + (http://bugs.python.org/issue14406) -- thanks Ralf Schmitt +- Added missing setUp() methods to several test classes + + +2.1.2 +===== + +- Fixed installation problem on Python 3.1 + + +2.1.1 +===== + +- Fixed missing 'concurrent' package declaration in setup.py + + +2.1 +=== + +- Moved the code from the 'futures' package to 'concurrent.futures' to provide + a drop in backport that matches the code in Python 3.2 standard library +- Deprecated the old 'futures' package + + +2.0 +=== + +- Changed implementation to match PEP 3148 + + +1.0 +=== + +Initial release. diff --git a/third_party/pythonfutures/LICENSE b/third_party/pythonfutures/LICENSE new file mode 100755 index 00000000..c430db0f --- /dev/null +++ b/third_party/pythonfutures/LICENSE @@ -0,0 +1,21 @@ +Copyright 2009 Brian Quinlan. All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + 2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY BRIAN QUINLAN "AS IS" AND ANY EXPRESS OR IMPLIED +WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT +HALL THE FREEBSD PROJECT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE +OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF +ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. \ No newline at end of file diff --git a/third_party/pythonfutures/concurrent/__init__.py b/third_party/pythonfutures/concurrent/__init__.py new file mode 100755 index 00000000..b36383a6 --- /dev/null +++ b/third_party/pythonfutures/concurrent/__init__.py @@ -0,0 +1,3 @@ +from pkgutil import extend_path + +__path__ = extend_path(__path__, __name__) diff --git a/third_party/pythonfutures/concurrent/futures/__init__.py b/third_party/pythonfutures/concurrent/futures/__init__.py new file mode 100755 index 00000000..b5231f8a --- /dev/null +++ b/third_party/pythonfutures/concurrent/futures/__init__.py @@ -0,0 +1,18 @@ +# Copyright 2009 Brian Quinlan. All Rights Reserved. +# Licensed to PSF under a Contributor Agreement. + +"""Execute computations asynchronously using threads or processes.""" + +__author__ = 'Brian Quinlan (brian@sweetapp.com)' + +from concurrent.futures._base import (FIRST_COMPLETED, + FIRST_EXCEPTION, + ALL_COMPLETED, + CancelledError, + TimeoutError, + Future, + Executor, + wait, + as_completed) +from concurrent.futures.process import ProcessPoolExecutor +from concurrent.futures.thread import ThreadPoolExecutor diff --git a/third_party/pythonfutures/concurrent/futures/_base.py b/third_party/pythonfutures/concurrent/futures/_base.py new file mode 100755 index 00000000..8ed69b7d --- /dev/null +++ b/third_party/pythonfutures/concurrent/futures/_base.py @@ -0,0 +1,574 @@ +# Copyright 2009 Brian Quinlan. All Rights Reserved. +# Licensed to PSF under a Contributor Agreement. + +from __future__ import with_statement +import logging +import threading +import time + +try: + from collections import namedtuple +except ImportError: + from concurrent.futures._compat import namedtuple + +__author__ = 'Brian Quinlan (brian@sweetapp.com)' + +FIRST_COMPLETED = 'FIRST_COMPLETED' +FIRST_EXCEPTION = 'FIRST_EXCEPTION' +ALL_COMPLETED = 'ALL_COMPLETED' +_AS_COMPLETED = '_AS_COMPLETED' + +# Possible future states (for internal use by the futures package). +PENDING = 'PENDING' +RUNNING = 'RUNNING' +# The future was cancelled by the user... +CANCELLED = 'CANCELLED' +# ...and _Waiter.add_cancelled() was called by a worker. +CANCELLED_AND_NOTIFIED = 'CANCELLED_AND_NOTIFIED' +FINISHED = 'FINISHED' + +_FUTURE_STATES = [ + PENDING, + RUNNING, + CANCELLED, + CANCELLED_AND_NOTIFIED, + FINISHED +] + +_STATE_TO_DESCRIPTION_MAP = { + PENDING: "pending", + RUNNING: "running", + CANCELLED: "cancelled", + CANCELLED_AND_NOTIFIED: "cancelled", + FINISHED: "finished" +} + +# Logger for internal use by the futures package. +LOGGER = logging.getLogger("concurrent.futures") + +class Error(Exception): + """Base class for all future-related exceptions.""" + pass + +class CancelledError(Error): + """The Future was cancelled.""" + pass + +class TimeoutError(Error): + """The operation exceeded the given deadline.""" + pass + +class _Waiter(object): + """Provides the event that wait() and as_completed() block on.""" + def __init__(self): + self.event = threading.Event() + self.finished_futures = [] + + def add_result(self, future): + self.finished_futures.append(future) + + def add_exception(self, future): + self.finished_futures.append(future) + + def add_cancelled(self, future): + self.finished_futures.append(future) + +class _AsCompletedWaiter(_Waiter): + """Used by as_completed().""" + + def __init__(self): + super(_AsCompletedWaiter, self).__init__() + self.lock = threading.Lock() + + def add_result(self, future): + with self.lock: + super(_AsCompletedWaiter, self).add_result(future) + self.event.set() + + def add_exception(self, future): + with self.lock: + super(_AsCompletedWaiter, self).add_exception(future) + self.event.set() + + def add_cancelled(self, future): + with self.lock: + super(_AsCompletedWaiter, self).add_cancelled(future) + self.event.set() + +class _FirstCompletedWaiter(_Waiter): + """Used by wait(return_when=FIRST_COMPLETED).""" + + def add_result(self, future): + super(_FirstCompletedWaiter, self).add_result(future) + self.event.set() + + def add_exception(self, future): + super(_FirstCompletedWaiter, self).add_exception(future) + self.event.set() + + def add_cancelled(self, future): + super(_FirstCompletedWaiter, self).add_cancelled(future) + self.event.set() + +class _AllCompletedWaiter(_Waiter): + """Used by wait(return_when=FIRST_EXCEPTION and ALL_COMPLETED).""" + + def __init__(self, num_pending_calls, stop_on_exception): + self.num_pending_calls = num_pending_calls + self.stop_on_exception = stop_on_exception + self.lock = threading.Lock() + super(_AllCompletedWaiter, self).__init__() + + def _decrement_pending_calls(self): + with self.lock: + self.num_pending_calls -= 1 + if not self.num_pending_calls: + self.event.set() + + def add_result(self, future): + super(_AllCompletedWaiter, self).add_result(future) + self._decrement_pending_calls() + + def add_exception(self, future): + super(_AllCompletedWaiter, self).add_exception(future) + if self.stop_on_exception: + self.event.set() + else: + self._decrement_pending_calls() + + def add_cancelled(self, future): + super(_AllCompletedWaiter, self).add_cancelled(future) + self._decrement_pending_calls() + +class _AcquireFutures(object): + """A context manager that does an ordered acquire of Future conditions.""" + + def __init__(self, futures): + self.futures = sorted(futures, key=id) + + def __enter__(self): + for future in self.futures: + future._condition.acquire() + + def __exit__(self, *args): + for future in self.futures: + future._condition.release() + +def _create_and_install_waiters(fs, return_when): + if return_when == _AS_COMPLETED: + waiter = _AsCompletedWaiter() + elif return_when == FIRST_COMPLETED: + waiter = _FirstCompletedWaiter() + else: + pending_count = sum( + f._state not in [CANCELLED_AND_NOTIFIED, FINISHED] for f in fs) + + if return_when == FIRST_EXCEPTION: + waiter = _AllCompletedWaiter(pending_count, stop_on_exception=True) + elif return_when == ALL_COMPLETED: + waiter = _AllCompletedWaiter(pending_count, stop_on_exception=False) + else: + raise ValueError("Invalid return condition: %r" % return_when) + + for f in fs: + f._waiters.append(waiter) + + return waiter + +def as_completed(fs, timeout=None): + """An iterator over the given futures that yields each as it completes. + + Args: + fs: The sequence of Futures (possibly created by different Executors) to + iterate over. + timeout: The maximum number of seconds to wait. If None, then there + is no limit on the wait time. + + Returns: + An iterator that yields the given Futures as they complete (finished or + cancelled). + + Raises: + TimeoutError: If the entire result iterator could not be generated + before the given timeout. + """ + if timeout is not None: + end_time = timeout + time.time() + + with _AcquireFutures(fs): + finished = set( + f for f in fs + if f._state in [CANCELLED_AND_NOTIFIED, FINISHED]) + pending = set(fs) - finished + waiter = _create_and_install_waiters(fs, _AS_COMPLETED) + + try: + for future in finished: + yield future + + while pending: + if timeout is None: + wait_timeout = None + else: + wait_timeout = end_time - time.time() + if wait_timeout < 0: + raise TimeoutError( + '%d (of %d) futures unfinished' % ( + len(pending), len(fs))) + + waiter.event.wait(wait_timeout) + + with waiter.lock: + finished = waiter.finished_futures + waiter.finished_futures = [] + waiter.event.clear() + + for future in finished: + yield future + pending.remove(future) + + finally: + for f in fs: + f._waiters.remove(waiter) + +DoneAndNotDoneFutures = namedtuple( + 'DoneAndNotDoneFutures', 'done not_done') +def wait(fs, timeout=None, return_when=ALL_COMPLETED): + """Wait for the futures in the given sequence to complete. + + Args: + fs: The sequence of Futures (possibly created by different Executors) to + wait upon. + timeout: The maximum number of seconds to wait. If None, then there + is no limit on the wait time. + return_when: Indicates when this function should return. The options + are: + + FIRST_COMPLETED - Return when any future finishes or is + cancelled. + FIRST_EXCEPTION - Return when any future finishes by raising an + exception. If no future raises an exception + then it is equivalent to ALL_COMPLETED. + ALL_COMPLETED - Return when all futures finish or are cancelled. + + Returns: + A named 2-tuple of sets. The first set, named 'done', contains the + futures that completed (is finished or cancelled) before the wait + completed. The second set, named 'not_done', contains uncompleted + futures. + """ + with _AcquireFutures(fs): + done = set(f for f in fs + if f._state in [CANCELLED_AND_NOTIFIED, FINISHED]) + not_done = set(fs) - done + + if (return_when == FIRST_COMPLETED) and done: + return DoneAndNotDoneFutures(done, not_done) + elif (return_when == FIRST_EXCEPTION) and done: + if any(f for f in done + if not f.cancelled() and f.exception() is not None): + return DoneAndNotDoneFutures(done, not_done) + + if len(done) == len(fs): + return DoneAndNotDoneFutures(done, not_done) + + waiter = _create_and_install_waiters(fs, return_when) + + waiter.event.wait(timeout) + for f in fs: + f._waiters.remove(waiter) + + done.update(waiter.finished_futures) + return DoneAndNotDoneFutures(done, set(fs) - done) + +class Future(object): + """Represents the result of an asynchronous computation.""" + + def __init__(self): + """Initializes the future. Should not be called by clients.""" + self._condition = threading.Condition() + self._state = PENDING + self._result = None + self._exception = None + self._waiters = [] + self._done_callbacks = [] + + def _invoke_callbacks(self): + for callback in self._done_callbacks: + try: + callback(self) + except Exception: + LOGGER.exception('exception calling callback for %r', self) + + def __repr__(self): + with self._condition: + if self._state == FINISHED: + if self._exception: + return '' % ( + hex(id(self)), + _STATE_TO_DESCRIPTION_MAP[self._state], + self._exception.__class__.__name__) + else: + return '' % ( + hex(id(self)), + _STATE_TO_DESCRIPTION_MAP[self._state], + self._result.__class__.__name__) + return '' % ( + hex(id(self)), + _STATE_TO_DESCRIPTION_MAP[self._state]) + + def cancel(self): + """Cancel the future if possible. + + Returns True if the future was cancelled, False otherwise. A future + cannot be cancelled if it is running or has already completed. + """ + with self._condition: + if self._state in [RUNNING, FINISHED]: + return False + + if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: + return True + + self._state = CANCELLED + self._condition.notify_all() + + self._invoke_callbacks() + return True + + def cancelled(self): + """Return True if the future has cancelled.""" + with self._condition: + return self._state in [CANCELLED, CANCELLED_AND_NOTIFIED] + + def running(self): + """Return True if the future is currently executing.""" + with self._condition: + return self._state == RUNNING + + def done(self): + """Return True of the future was cancelled or finished executing.""" + with self._condition: + return self._state in [CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED] + + def __get_result(self): + if self._exception: + raise self._exception + else: + return self._result + + def add_done_callback(self, fn): + """Attaches a callable that will be called when the future finishes. + + Args: + fn: A callable that will be called with this future as its only + argument when the future completes or is cancelled. The callable + will always be called by a thread in the same process in which + it was added. If the future has already completed or been + cancelled then the callable will be called immediately. These + callables are called in the order that they were added. + """ + with self._condition: + if self._state not in [CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED]: + self._done_callbacks.append(fn) + return + fn(self) + + def result(self, timeout=None): + """Return the result of the call that the future represents. + + Args: + timeout: The number of seconds to wait for the result if the future + isn't done. If None, then there is no limit on the wait time. + + Returns: + The result of the call that the future represents. + + Raises: + CancelledError: If the future was cancelled. + TimeoutError: If the future didn't finish executing before the given + timeout. + Exception: If the call raised then that exception will be raised. + """ + with self._condition: + if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: + raise CancelledError() + elif self._state == FINISHED: + return self.__get_result() + + self._condition.wait(timeout) + + if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: + raise CancelledError() + elif self._state == FINISHED: + return self.__get_result() + else: + raise TimeoutError() + + def exception(self, timeout=None): + """Return the exception raised by the call that the future represents. + + Args: + timeout: The number of seconds to wait for the exception if the + future isn't done. If None, then there is no limit on the wait + time. + + Returns: + The exception raised by the call that the future represents or None + if the call completed without raising. + + Raises: + CancelledError: If the future was cancelled. + TimeoutError: If the future didn't finish executing before the given + timeout. + """ + + with self._condition: + if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: + raise CancelledError() + elif self._state == FINISHED: + return self._exception + + self._condition.wait(timeout) + + if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: + raise CancelledError() + elif self._state == FINISHED: + return self._exception + else: + raise TimeoutError() + + # The following methods should only be used by Executors and in tests. + def set_running_or_notify_cancel(self): + """Mark the future as running or process any cancel notifications. + + Should only be used by Executor implementations and unit tests. + + If the future has been cancelled (cancel() was called and returned + True) then any threads waiting on the future completing (though calls + to as_completed() or wait()) are notified and False is returned. + + If the future was not cancelled then it is put in the running state + (future calls to running() will return True) and True is returned. + + This method should be called by Executor implementations before + executing the work associated with this future. If this method returns + False then the work should not be executed. + + Returns: + False if the Future was cancelled, True otherwise. + + Raises: + RuntimeError: if this method was already called or if set_result() + or set_exception() was called. + """ + with self._condition: + if self._state == CANCELLED: + self._state = CANCELLED_AND_NOTIFIED + for waiter in self._waiters: + waiter.add_cancelled(self) + # self._condition.notify_all() is not necessary because + # self.cancel() triggers a notification. + return False + elif self._state == PENDING: + self._state = RUNNING + return True + else: + LOGGER.critical('Future %s in unexpected state: %s', + id(self.future), + self.future._state) + raise RuntimeError('Future in unexpected state') + + def set_result(self, result): + """Sets the return value of work associated with the future. + + Should only be used by Executor implementations and unit tests. + """ + with self._condition: + self._result = result + self._state = FINISHED + for waiter in self._waiters: + waiter.add_result(self) + self._condition.notify_all() + self._invoke_callbacks() + + def set_exception(self, exception): + """Sets the result of the future as being the given exception. + + Should only be used by Executor implementations and unit tests. + """ + with self._condition: + self._exception = exception + self._state = FINISHED + for waiter in self._waiters: + waiter.add_exception(self) + self._condition.notify_all() + self._invoke_callbacks() + +class Executor(object): + """This is an abstract base class for concrete asynchronous executors.""" + + def submit(self, fn, *args, **kwargs): + """Submits a callable to be executed with the given arguments. + + Schedules the callable to be executed as fn(*args, **kwargs) and returns + a Future instance representing the execution of the callable. + + Returns: + A Future representing the given call. + """ + raise NotImplementedError() + + def map(self, fn, *iterables, **kwargs): + """Returns a iterator equivalent to map(fn, iter). + + Args: + fn: A callable that will take as many arguments as there are + passed iterables. + timeout: The maximum number of seconds to wait. If None, then there + is no limit on the wait time. + + Returns: + An iterator equivalent to: map(func, *iterables) but the calls may + be evaluated out-of-order. + + Raises: + TimeoutError: If the entire result iterator could not be generated + before the given timeout. + Exception: If fn(*args) raises for any values. + """ + timeout = kwargs.get('timeout') + if timeout is not None: + end_time = timeout + time.time() + + fs = [self.submit(fn, *args) for args in zip(*iterables)] + + try: + for future in fs: + if timeout is None: + yield future.result() + else: + yield future.result(end_time - time.time()) + finally: + for future in fs: + future.cancel() + + def shutdown(self, wait=True): + """Clean-up the resources associated with the Executor. + + It is safe to call this method several times. Otherwise, no other + methods can be called after this one. + + Args: + wait: If True then shutdown will not return until all running + futures have finished executing and the resources used by the + executor have been reclaimed. + """ + pass + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + self.shutdown(wait=True) + return False diff --git a/third_party/pythonfutures/concurrent/futures/_compat.py b/third_party/pythonfutures/concurrent/futures/_compat.py new file mode 100755 index 00000000..11462326 --- /dev/null +++ b/third_party/pythonfutures/concurrent/futures/_compat.py @@ -0,0 +1,101 @@ +from keyword import iskeyword as _iskeyword +from operator import itemgetter as _itemgetter +import sys as _sys + + +def namedtuple(typename, field_names): + """Returns a new subclass of tuple with named fields. + + >>> Point = namedtuple('Point', 'x y') + >>> Point.__doc__ # docstring for the new class + 'Point(x, y)' + >>> p = Point(11, y=22) # instantiate with positional args or keywords + >>> p[0] + p[1] # indexable like a plain tuple + 33 + >>> x, y = p # unpack like a regular tuple + >>> x, y + (11, 22) + >>> p.x + p.y # fields also accessable by name + 33 + >>> d = p._asdict() # convert to a dictionary + >>> d['x'] + 11 + >>> Point(**d) # convert from a dictionary + Point(x=11, y=22) + >>> p._replace(x=100) # _replace() is like str.replace() but targets named fields + Point(x=100, y=22) + + """ + + # Parse and validate the field names. Validation serves two purposes, + # generating informative error messages and preventing template injection attacks. + if isinstance(field_names, basestring): + field_names = field_names.replace(',', ' ').split() # names separated by whitespace and/or commas + field_names = tuple(map(str, field_names)) + for name in (typename,) + field_names: + if not all(c.isalnum() or c=='_' for c in name): + raise ValueError('Type names and field names can only contain alphanumeric characters and underscores: %r' % name) + if _iskeyword(name): + raise ValueError('Type names and field names cannot be a keyword: %r' % name) + if name[0].isdigit(): + raise ValueError('Type names and field names cannot start with a number: %r' % name) + seen_names = set() + for name in field_names: + if name.startswith('_'): + raise ValueError('Field names cannot start with an underscore: %r' % name) + if name in seen_names: + raise ValueError('Encountered duplicate field name: %r' % name) + seen_names.add(name) + + # Create and fill-in the class template + numfields = len(field_names) + argtxt = repr(field_names).replace("'", "")[1:-1] # tuple repr without parens or quotes + reprtxt = ', '.join('%s=%%r' % name for name in field_names) + dicttxt = ', '.join('%r: t[%d]' % (name, pos) for pos, name in enumerate(field_names)) + template = '''class %(typename)s(tuple): + '%(typename)s(%(argtxt)s)' \n + __slots__ = () \n + _fields = %(field_names)r \n + def __new__(_cls, %(argtxt)s): + return _tuple.__new__(_cls, (%(argtxt)s)) \n + @classmethod + def _make(cls, iterable, new=tuple.__new__, len=len): + 'Make a new %(typename)s object from a sequence or iterable' + result = new(cls, iterable) + if len(result) != %(numfields)d: + raise TypeError('Expected %(numfields)d arguments, got %%d' %% len(result)) + return result \n + def __repr__(self): + return '%(typename)s(%(reprtxt)s)' %% self \n + def _asdict(t): + 'Return a new dict which maps field names to their values' + return {%(dicttxt)s} \n + def _replace(_self, **kwds): + 'Return a new %(typename)s object replacing specified fields with new values' + result = _self._make(map(kwds.pop, %(field_names)r, _self)) + if kwds: + raise ValueError('Got unexpected field names: %%r' %% kwds.keys()) + return result \n + def __getnewargs__(self): + return tuple(self) \n\n''' % locals() + for i, name in enumerate(field_names): + template += ' %s = _property(_itemgetter(%d))\n' % (name, i) + + # Execute the template string in a temporary namespace and + # support tracing utilities by setting a value for frame.f_globals['__name__'] + namespace = dict(_itemgetter=_itemgetter, __name__='namedtuple_%s' % typename, + _property=property, _tuple=tuple) + try: + exec(template, namespace) + except SyntaxError: + e = _sys.exc_info()[1] + raise SyntaxError(e.message + ':\n' + template) + result = namespace[typename] + + # For pickling to work, the __module__ variable needs to be set to the frame + # where the named tuple is created. Bypass this step in enviroments where + # sys._getframe is not defined (Jython for example). + if hasattr(_sys, '_getframe'): + result.__module__ = _sys._getframe(1).f_globals.get('__name__', '__main__') + + return result diff --git a/third_party/pythonfutures/concurrent/futures/process.py b/third_party/pythonfutures/concurrent/futures/process.py new file mode 100755 index 00000000..98684f8e --- /dev/null +++ b/third_party/pythonfutures/concurrent/futures/process.py @@ -0,0 +1,363 @@ +# Copyright 2009 Brian Quinlan. All Rights Reserved. +# Licensed to PSF under a Contributor Agreement. + +"""Implements ProcessPoolExecutor. + +The follow diagram and text describe the data-flow through the system: + +|======================= In-process =====================|== Out-of-process ==| + ++----------+ +----------+ +--------+ +-----------+ +---------+ +| | => | Work Ids | => | | => | Call Q | => | | +| | +----------+ | | +-----------+ | | +| | | ... | | | | ... | | | +| | | 6 | | | | 5, call() | | | +| | | 7 | | | | ... | | | +| Process | | ... | | Local | +-----------+ | Process | +| Pool | +----------+ | Worker | | #1..n | +| Executor | | Thread | | | +| | +----------- + | | +-----------+ | | +| | <=> | Work Items | <=> | | <= | Result Q | <= | | +| | +------------+ | | +-----------+ | | +| | | 6: call() | | | | ... | | | +| | | future | | | | 4, result | | | +| | | ... | | | | 3, except | | | ++----------+ +------------+ +--------+ +-----------+ +---------+ + +Executor.submit() called: +- creates a uniquely numbered _WorkItem and adds it to the "Work Items" dict +- adds the id of the _WorkItem to the "Work Ids" queue + +Local worker thread: +- reads work ids from the "Work Ids" queue and looks up the corresponding + WorkItem from the "Work Items" dict: if the work item has been cancelled then + it is simply removed from the dict, otherwise it is repackaged as a + _CallItem and put in the "Call Q". New _CallItems are put in the "Call Q" + until "Call Q" is full. NOTE: the size of the "Call Q" is kept small because + calls placed in the "Call Q" can no longer be cancelled with Future.cancel(). +- reads _ResultItems from "Result Q", updates the future stored in the + "Work Items" dict and deletes the dict entry + +Process #1..n: +- reads _CallItems from "Call Q", executes the calls, and puts the resulting + _ResultItems in "Request Q" +""" + +from __future__ import with_statement +import atexit +import multiprocessing +import threading +import weakref +import sys + +from concurrent.futures import _base + +try: + import queue +except ImportError: + import Queue as queue + +__author__ = 'Brian Quinlan (brian@sweetapp.com)' + +# Workers are created as daemon threads and processes. This is done to allow the +# interpreter to exit when there are still idle processes in a +# ProcessPoolExecutor's process pool (i.e. shutdown() was not called). However, +# allowing workers to die with the interpreter has two undesirable properties: +# - The workers would still be running during interpretor shutdown, +# meaning that they would fail in unpredictable ways. +# - The workers could be killed while evaluating a work item, which could +# be bad if the callable being evaluated has external side-effects e.g. +# writing to a file. +# +# To work around this problem, an exit handler is installed which tells the +# workers to exit when their work queues are empty and then waits until the +# threads/processes finish. + +_threads_queues = weakref.WeakKeyDictionary() +_shutdown = False + +def _python_exit(): + global _shutdown + _shutdown = True + items = list(_threads_queues.items()) + for t, q in items: + q.put(None) + for t, q in items: + t.join() + +# Controls how many more calls than processes will be queued in the call queue. +# A smaller number will mean that processes spend more time idle waiting for +# work while a larger number will make Future.cancel() succeed less frequently +# (Futures in the call queue cannot be cancelled). +EXTRA_QUEUED_CALLS = 1 + +class _WorkItem(object): + def __init__(self, future, fn, args, kwargs): + self.future = future + self.fn = fn + self.args = args + self.kwargs = kwargs + +class _ResultItem(object): + def __init__(self, work_id, exception=None, result=None): + self.work_id = work_id + self.exception = exception + self.result = result + +class _CallItem(object): + def __init__(self, work_id, fn, args, kwargs): + self.work_id = work_id + self.fn = fn + self.args = args + self.kwargs = kwargs + +def _process_worker(call_queue, result_queue): + """Evaluates calls from call_queue and places the results in result_queue. + + This worker is run in a separate process. + + Args: + call_queue: A multiprocessing.Queue of _CallItems that will be read and + evaluated by the worker. + result_queue: A multiprocessing.Queue of _ResultItems that will written + to by the worker. + shutdown: A multiprocessing.Event that will be set as a signal to the + worker that it should exit when call_queue is empty. + """ + while True: + call_item = call_queue.get(block=True) + if call_item is None: + # Wake up queue management thread + result_queue.put(None) + return + try: + r = call_item.fn(*call_item.args, **call_item.kwargs) + except BaseException: + e = sys.exc_info()[1] + result_queue.put(_ResultItem(call_item.work_id, + exception=e)) + else: + result_queue.put(_ResultItem(call_item.work_id, + result=r)) + +def _add_call_item_to_queue(pending_work_items, + work_ids, + call_queue): + """Fills call_queue with _WorkItems from pending_work_items. + + This function never blocks. + + Args: + pending_work_items: A dict mapping work ids to _WorkItems e.g. + {5: <_WorkItem...>, 6: <_WorkItem...>, ...} + work_ids: A queue.Queue of work ids e.g. Queue([5, 6, ...]). Work ids + are consumed and the corresponding _WorkItems from + pending_work_items are transformed into _CallItems and put in + call_queue. + call_queue: A multiprocessing.Queue that will be filled with _CallItems + derived from _WorkItems. + """ + while True: + if call_queue.full(): + return + try: + work_id = work_ids.get(block=False) + except queue.Empty: + return + else: + work_item = pending_work_items[work_id] + + if work_item.future.set_running_or_notify_cancel(): + call_queue.put(_CallItem(work_id, + work_item.fn, + work_item.args, + work_item.kwargs), + block=True) + else: + del pending_work_items[work_id] + continue + +def _queue_management_worker(executor_reference, + processes, + pending_work_items, + work_ids_queue, + call_queue, + result_queue): + """Manages the communication between this process and the worker processes. + + This function is run in a local thread. + + Args: + executor_reference: A weakref.ref to the ProcessPoolExecutor that owns + this thread. Used to determine if the ProcessPoolExecutor has been + garbage collected and that this function can exit. + process: A list of the multiprocessing.Process instances used as + workers. + pending_work_items: A dict mapping work ids to _WorkItems e.g. + {5: <_WorkItem...>, 6: <_WorkItem...>, ...} + work_ids_queue: A queue.Queue of work ids e.g. Queue([5, 6, ...]). + call_queue: A multiprocessing.Queue that will be filled with _CallItems + derived from _WorkItems for processing by the process workers. + result_queue: A multiprocessing.Queue of _ResultItems generated by the + process workers. + """ + nb_shutdown_processes = [0] + def shutdown_one_process(): + """Tell a worker to terminate, which will in turn wake us again""" + call_queue.put(None) + nb_shutdown_processes[0] += 1 + while True: + _add_call_item_to_queue(pending_work_items, + work_ids_queue, + call_queue) + + result_item = result_queue.get(block=True) + if result_item is not None: + work_item = pending_work_items[result_item.work_id] + del pending_work_items[result_item.work_id] + + if result_item.exception: + work_item.future.set_exception(result_item.exception) + else: + work_item.future.set_result(result_item.result) + # Check whether we should start shutting down. + executor = executor_reference() + # No more work items can be added if: + # - The interpreter is shutting down OR + # - The executor that owns this worker has been collected OR + # - The executor that owns this worker has been shutdown. + if _shutdown or executor is None or executor._shutdown_thread: + # Since no new work items can be added, it is safe to shutdown + # this thread if there are no pending work items. + if not pending_work_items: + while nb_shutdown_processes[0] < len(processes): + shutdown_one_process() + # If .join() is not called on the created processes then + # some multiprocessing.Queue methods may deadlock on Mac OS + # X. + for p in processes: + p.join() + call_queue.close() + return + del executor + +_system_limits_checked = False +_system_limited = None +def _check_system_limits(): + global _system_limits_checked, _system_limited + if _system_limits_checked: + if _system_limited: + raise NotImplementedError(_system_limited) + _system_limits_checked = True + try: + import os + nsems_max = os.sysconf("SC_SEM_NSEMS_MAX") + except (AttributeError, ValueError): + # sysconf not available or setting not available + return + if nsems_max == -1: + # indetermine limit, assume that limit is determined + # by available memory only + return + if nsems_max >= 256: + # minimum number of semaphores available + # according to POSIX + return + _system_limited = "system provides too few semaphores (%d available, 256 necessary)" % nsems_max + raise NotImplementedError(_system_limited) + +class ProcessPoolExecutor(_base.Executor): + def __init__(self, max_workers=None): + """Initializes a new ProcessPoolExecutor instance. + + Args: + max_workers: The maximum number of processes that can be used to + execute the given calls. If None or not given then as many + worker processes will be created as the machine has processors. + """ + _check_system_limits() + + if max_workers is None: + self._max_workers = multiprocessing.cpu_count() + else: + self._max_workers = max_workers + + # Make the call queue slightly larger than the number of processes to + # prevent the worker processes from idling. But don't make it too big + # because futures in the call queue cannot be cancelled. + self._call_queue = multiprocessing.Queue(self._max_workers + + EXTRA_QUEUED_CALLS) + self._result_queue = multiprocessing.Queue() + self._work_ids = queue.Queue() + self._queue_management_thread = None + self._processes = set() + + # Shutdown is a two-step process. + self._shutdown_thread = False + self._shutdown_lock = threading.Lock() + self._queue_count = 0 + self._pending_work_items = {} + + def _start_queue_management_thread(self): + # When the executor gets lost, the weakref callback will wake up + # the queue management thread. + def weakref_cb(_, q=self._result_queue): + q.put(None) + if self._queue_management_thread is None: + self._queue_management_thread = threading.Thread( + target=_queue_management_worker, + args=(weakref.ref(self, weakref_cb), + self._processes, + self._pending_work_items, + self._work_ids, + self._call_queue, + self._result_queue)) + self._queue_management_thread.daemon = True + self._queue_management_thread.start() + _threads_queues[self._queue_management_thread] = self._result_queue + + def _adjust_process_count(self): + for _ in range(len(self._processes), self._max_workers): + p = multiprocessing.Process( + target=_process_worker, + args=(self._call_queue, + self._result_queue)) + p.start() + self._processes.add(p) + + def submit(self, fn, *args, **kwargs): + with self._shutdown_lock: + if self._shutdown_thread: + raise RuntimeError('cannot schedule new futures after shutdown') + + f = _base.Future() + w = _WorkItem(f, fn, args, kwargs) + + self._pending_work_items[self._queue_count] = w + self._work_ids.put(self._queue_count) + self._queue_count += 1 + # Wake up queue management thread + self._result_queue.put(None) + + self._start_queue_management_thread() + self._adjust_process_count() + return f + submit.__doc__ = _base.Executor.submit.__doc__ + + def shutdown(self, wait=True): + with self._shutdown_lock: + self._shutdown_thread = True + if self._queue_management_thread: + # Wake up queue management thread + self._result_queue.put(None) + if wait: + self._queue_management_thread.join() + # To reduce the risk of openning too many files, remove references to + # objects that use file descriptors. + self._queue_management_thread = None + self._call_queue = None + self._result_queue = None + self._processes = None + shutdown.__doc__ = _base.Executor.shutdown.__doc__ + +atexit.register(_python_exit) diff --git a/third_party/pythonfutures/concurrent/futures/thread.py b/third_party/pythonfutures/concurrent/futures/thread.py new file mode 100755 index 00000000..a45959d3 --- /dev/null +++ b/third_party/pythonfutures/concurrent/futures/thread.py @@ -0,0 +1,138 @@ +# Copyright 2009 Brian Quinlan. All Rights Reserved. +# Licensed to PSF under a Contributor Agreement. + +"""Implements ThreadPoolExecutor.""" + +from __future__ import with_statement +import atexit +import threading +import weakref +import sys + +from concurrent.futures import _base + +try: + import queue +except ImportError: + import Queue as queue + +__author__ = 'Brian Quinlan (brian@sweetapp.com)' + +# Workers are created as daemon threads. This is done to allow the interpreter +# to exit when there are still idle threads in a ThreadPoolExecutor's thread +# pool (i.e. shutdown() was not called). However, allowing workers to die with +# the interpreter has two undesirable properties: +# - The workers would still be running during interpretor shutdown, +# meaning that they would fail in unpredictable ways. +# - The workers could be killed while evaluating a work item, which could +# be bad if the callable being evaluated has external side-effects e.g. +# writing to a file. +# +# To work around this problem, an exit handler is installed which tells the +# workers to exit when their work queues are empty and then waits until the +# threads finish. + +_threads_queues = weakref.WeakKeyDictionary() +_shutdown = False + +def _python_exit(): + global _shutdown + _shutdown = True + items = list(_threads_queues.items()) + for t, q in items: + q.put(None) + for t, q in items: + t.join() + +atexit.register(_python_exit) + +class _WorkItem(object): + def __init__(self, future, fn, args, kwargs): + self.future = future + self.fn = fn + self.args = args + self.kwargs = kwargs + + def run(self): + if not self.future.set_running_or_notify_cancel(): + return + + try: + result = self.fn(*self.args, **self.kwargs) + except BaseException: + e = sys.exc_info()[1] + self.future.set_exception(e) + else: + self.future.set_result(result) + +def _worker(executor_reference, work_queue): + try: + while True: + work_item = work_queue.get(block=True) + if work_item is not None: + work_item.run() + continue + executor = executor_reference() + # Exit if: + # - The interpreter is shutting down OR + # - The executor that owns the worker has been collected OR + # - The executor that owns the worker has been shutdown. + if _shutdown or executor is None or executor._shutdown: + # Notice other workers + work_queue.put(None) + return + del executor + except BaseException: + _base.LOGGER.critical('Exception in worker', exc_info=True) + +class ThreadPoolExecutor(_base.Executor): + def __init__(self, max_workers): + """Initializes a new ThreadPoolExecutor instance. + + Args: + max_workers: The maximum number of threads that can be used to + execute the given calls. + """ + self._max_workers = max_workers + self._work_queue = queue.Queue() + self._threads = set() + self._shutdown = False + self._shutdown_lock = threading.Lock() + + def submit(self, fn, *args, **kwargs): + with self._shutdown_lock: + if self._shutdown: + raise RuntimeError('cannot schedule new futures after shutdown') + + f = _base.Future() + w = _WorkItem(f, fn, args, kwargs) + + self._work_queue.put(w) + self._adjust_thread_count() + return f + submit.__doc__ = _base.Executor.submit.__doc__ + + def _adjust_thread_count(self): + # When the executor gets lost, the weakref callback will wake up + # the worker threads. + def weakref_cb(_, q=self._work_queue): + q.put(None) + # TODO(bquinlan): Should avoid creating new threads if there are more + # idle threads than items in the work queue. + if len(self._threads) < self._max_workers: + t = threading.Thread(target=_worker, + args=(weakref.ref(self, weakref_cb), + self._work_queue)) + t.daemon = True + t.start() + self._threads.add(t) + _threads_queues[t] = self._work_queue + + def shutdown(self, wait=True): + with self._shutdown_lock: + self._shutdown = True + self._work_queue.put(None) + if wait: + for t in self._threads: + t.join() + shutdown.__doc__ = _base.Executor.shutdown.__doc__ diff --git a/third_party/pythonfutures/crawl.py b/third_party/pythonfutures/crawl.py new file mode 100755 index 00000000..86e0af7f --- /dev/null +++ b/third_party/pythonfutures/crawl.py @@ -0,0 +1,74 @@ +"""Compare the speed of downloading URLs sequentially vs. using futures.""" + +import functools +import time +import timeit +import sys + +try: + from urllib2 import urlopen +except ImportError: + from urllib.request import urlopen + +from concurrent.futures import (as_completed, ThreadPoolExecutor, + ProcessPoolExecutor) + +URLS = ['http://www.google.com/', + 'http://www.apple.com/', + 'http://www.ibm.com', + 'http://www.thisurlprobablydoesnotexist.com', + 'http://www.slashdot.org/', + 'http://www.python.org/', + 'http://www.bing.com/', + 'http://www.facebook.com/', + 'http://www.yahoo.com/', + 'http://www.youtube.com/', + 'http://www.blogger.com/'] + +def load_url(url, timeout): + kwargs = {'timeout': timeout} if sys.version_info >= (2, 6) else {} + return urlopen(url, **kwargs).read() + +def download_urls_sequential(urls, timeout=60): + url_to_content = {} + for url in urls: + try: + url_to_content[url] = load_url(url, timeout=timeout) + except: + pass + return url_to_content + +def download_urls_with_executor(urls, executor, timeout=60): + try: + url_to_content = {} + future_to_url = dict((executor.submit(load_url, url, timeout), url) + for url in urls) + + for future in as_completed(future_to_url): + try: + url_to_content[future_to_url[future]] = future.result() + except: + pass + return url_to_content + finally: + executor.shutdown() + +def main(): + for name, fn in [('sequential', + functools.partial(download_urls_sequential, URLS)), + ('processes', + functools.partial(download_urls_with_executor, + URLS, + ProcessPoolExecutor(10))), + ('threads', + functools.partial(download_urls_with_executor, + URLS, + ThreadPoolExecutor(10)))]: + sys.stdout.write('%s: ' % name.ljust(12)) + start = time.time() + url_map = fn() + sys.stdout.write('%.2f seconds (%d of %d downloaded)\n' % + (time.time() - start, len(url_map), len(URLS))) + +if __name__ == '__main__': + main() diff --git a/third_party/pythonfutures/docs/conf.py b/third_party/pythonfutures/docs/conf.py new file mode 100755 index 00000000..124cd518 --- /dev/null +++ b/third_party/pythonfutures/docs/conf.py @@ -0,0 +1,194 @@ +# -*- coding: utf-8 -*- +# +# futures documentation build configuration file, created by +# sphinx-quickstart on Wed Jun 3 19:35:34 2009. +# +# This file is execfile()d with the current directory set to its containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +import sys, os + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +#sys.path.append(os.path.abspath('.')) + +# -- General configuration ----------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be extensions +# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix of source filenames. +source_suffix = '.rst' + +# The encoding of source files. +#source_encoding = 'utf-8' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'futures' +copyright = u'2009-2011, Brian Quinlan' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = '2.1.3' +# The full version, including alpha/beta/rc tags. +release = '2.1.3' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +#language = None + +# There are two options for replacing |today|: either, you set today to some +# non-false value, then it is used: +#today = '' +# Else, today_fmt is used as the format for a strftime call. +#today_fmt = '%B %d, %Y' + +# List of documents that shouldn't be included in the build. +#unused_docs = [] + +# List of directories, relative to source directory, that shouldn't be searched +# for source files. +exclude_trees = ['_build'] + +# The reST default role (used for this markup: `text`) to use for all documents. +#default_role = None + +# If true, '()' will be appended to :func: etc. cross-reference text. +#add_function_parentheses = True + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +#add_module_names = True + +# If true, sectionauthor and moduleauthor directives will be shown in the +# output. They are ignored by default. +#show_authors = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# A list of ignored prefixes for module index sorting. +#modindex_common_prefix = [] + + +# -- Options for HTML output --------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. Major themes that come with +# Sphinx are currently 'default' and 'sphinxdoc'. +html_theme = 'default' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +#html_theme_options = {} + +# Add any paths that contain custom themes here, relative to this directory. +#html_theme_path = [] + +# The name for this set of Sphinx documents. If None, it defaults to +# " v documentation". +#html_title = None + +# A shorter title for the navigation bar. Default is the same as html_title. +#html_short_title = None + +# The name of an image file (relative to this directory) to place at the top +# of the sidebar. +#html_logo = None + +# The name of an image file (within the static path) to use as favicon of the +# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 +# pixels large. +#html_favicon = None + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, +# using the given strftime format. +#html_last_updated_fmt = '%b %d, %Y' + +# If true, SmartyPants will be used to convert quotes and dashes to +# typographically correct entities. +#html_use_smartypants = True + +# Custom sidebar templates, maps document names to template names. +#html_sidebars = {} + +# Additional templates that should be rendered to pages, maps page names to +# template names. +#html_additional_pages = {} + +# If false, no module index is generated. +#html_use_modindex = True + +# If false, no index is generated. +#html_use_index = True + +# If true, the index is split into individual pages for each letter. +#html_split_index = False + +# If true, links to the reST sources are added to the pages. +#html_show_sourcelink = True + +# If true, an OpenSearch description file will be output, and all pages will +# contain a tag referring to it. The value of this option must be the +# base URL from which the finished HTML is served. +#html_use_opensearch = '' + +# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). +#html_file_suffix = '' + +# Output file base name for HTML help builder. +htmlhelp_basename = 'futuresdoc' + + +# -- Options for LaTeX output -------------------------------------------------- + +# The paper size ('letter' or 'a4'). +#latex_paper_size = 'letter' + +# The font size ('10pt', '11pt' or '12pt'). +#latex_font_size = '10pt' + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, author, documentclass [howto/manual]). +latex_documents = [ + ('index', 'futures.tex', u'futures Documentation', + u'Brian Quinlan', 'manual'), +] + +# The name of an image file (relative to this directory) to place at the top of +# the title page. +#latex_logo = None + +# For "manual" documents, if this is true, then toplevel headings are parts, +# not chapters. +#latex_use_parts = False + +# Additional stuff for the LaTeX preamble. +#latex_preamble = '' + +# Documents to append as an appendix to all manuals. +#latex_appendices = [] + +# If false, no module index is generated. +#latex_use_modindex = True diff --git a/third_party/pythonfutures/docs/index.rst b/third_party/pythonfutures/docs/index.rst new file mode 100755 index 00000000..525ce6ab --- /dev/null +++ b/third_party/pythonfutures/docs/index.rst @@ -0,0 +1,345 @@ +:mod:`concurrent.futures` --- Asynchronous computation +====================================================== + +.. module:: concurrent.futures + :synopsis: Execute computations asynchronously using threads or processes. + +The :mod:`concurrent.futures` module provides a high-level interface for +asynchronously executing callables. + +The asynchronous execution can be be performed by threads using +:class:`ThreadPoolExecutor` or seperate processes using +:class:`ProcessPoolExecutor`. Both implement the same interface, which is +defined by the abstract :class:`Executor` class. + +Executor Objects +---------------- + +:class:`Executor` is an abstract class that provides methods to execute calls +asynchronously. It should not be used directly, but through its two +subclasses: :class:`ThreadPoolExecutor` and :class:`ProcessPoolExecutor`. + +.. method:: Executor.submit(fn, *args, **kwargs) + + Schedules the callable to be executed as *fn*(*\*args*, *\*\*kwargs*) and + returns a :class:`Future` representing the execution of the callable. + +:: + + with ThreadPoolExecutor(max_workers=1) as executor: + future = executor.submit(pow, 323, 1235) + print(future.result()) + +.. method:: Executor.map(func, *iterables, timeout=None) + + Equivalent to map(*func*, *\*iterables*) but func is executed asynchronously + and several calls to *func* may be made concurrently. The returned iterator + raises a :exc:`TimeoutError` if :meth:`__next__()` is called and the result + isn't available after *timeout* seconds from the original call to + :meth:`map()`. *timeout* can be an int or float. If *timeout* is not + specified or ``None`` then there is no limit to the wait time. If a call + raises an exception then that exception will be raised when its value is + retrieved from the iterator. + +.. method:: Executor.shutdown(wait=True) + + Signal the executor that it should free any resources that it is using when + the currently pending futures are done executing. Calls to + :meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will + raise :exc:`RuntimeError`. + + If *wait* is `True` then this method will not return until all the pending + futures are done executing and the resources associated with the executor + have been freed. If *wait* is `False` then this method will return + immediately and the resources associated with the executor will be freed + when all pending futures are done executing. Regardless of the value of + *wait*, the entire Python program will not exit until all pending futures + are done executing. + + You can avoid having to call this method explicitly if you use the `with` + statement, which will shutdown the `Executor` (waiting as if + `Executor.shutdown` were called with *wait* set to `True`): + +:: + + import shutil + with ThreadPoolExecutor(max_workers=4) as e: + e.submit(shutil.copy, 'src1.txt', 'dest1.txt') + e.submit(shutil.copy, 'src2.txt', 'dest2.txt') + e.submit(shutil.copy, 'src3.txt', 'dest3.txt') + e.submit(shutil.copy, 'src3.txt', 'dest4.txt') + + +ThreadPoolExecutor Objects +-------------------------- + +The :class:`ThreadPoolExecutor` class is an :class:`Executor` subclass that uses +a pool of threads to execute calls asynchronously. + +Deadlock can occur when the callable associated with a :class:`Future` waits on +the results of another :class:`Future`. For example: + +:: + + import time + def wait_on_b(): + time.sleep(5) + print(b.result()) # b will never complete because it is waiting on a. + return 5 + + def wait_on_a(): + time.sleep(5) + print(a.result()) # a will never complete because it is waiting on b. + return 6 + + + executor = ThreadPoolExecutor(max_workers=2) + a = executor.submit(wait_on_b) + b = executor.submit(wait_on_a) + +And: + +:: + + def wait_on_future(): + f = executor.submit(pow, 5, 2) + # This will never complete because there is only one worker thread and + # it is executing this function. + print(f.result()) + + executor = ThreadPoolExecutor(max_workers=1) + executor.submit(wait_on_future) + +.. class:: ThreadPoolExecutor(max_workers) + + Executes calls asynchronously using at pool of at most *max_workers* threads. + +.. _threadpoolexecutor-example: + +ThreadPoolExecutor Example +^^^^^^^^^^^^^^^^^^^^^^^^^^ +:: + + from concurrent import futures + import urllib.request + + URLS = ['http://www.foxnews.com/', + 'http://www.cnn.com/', + 'http://europe.wsj.com/', + 'http://www.bbc.co.uk/', + 'http://some-made-up-domain.com/'] + + def load_url(url, timeout): + return urllib.request.urlopen(url, timeout=timeout).read() + + with futures.ThreadPoolExecutor(max_workers=5) as executor: + future_to_url = dict((executor.submit(load_url, url, 60), url) + for url in URLS) + + for future in futures.as_completed(future_to_url): + url = future_to_url[future] + if future.exception() is not None: + print('%r generated an exception: %s' % (url, + future.exception())) + else: + print('%r page is %d bytes' % (url, len(future.result()))) + +ProcessPoolExecutor Objects +--------------------------- + +The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that +uses a pool of processes to execute calls asynchronously. +:class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which +allows it to side-step the :term:`Global Interpreter Lock` but also means that +only picklable objects can be executed and returned. + +Calling :class:`Executor` or :class:`Future` methods from a callable submitted +to a :class:`ProcessPoolExecutor` will result in deadlock. + +.. class:: ProcessPoolExecutor(max_workers=None) + + Executes calls asynchronously using a pool of at most *max_workers* + processes. If *max_workers* is ``None`` or not given then as many worker + processes will be created as the machine has processors. + +.. _processpoolexecutor-example: + +ProcessPoolExecutor Example +^^^^^^^^^^^^^^^^^^^^^^^^^^^ +:: + + import math + + PRIMES = [ + 112272535095293, + 112582705942171, + 112272535095293, + 115280095190773, + 115797848077099, + 1099726899285419] + + def is_prime(n): + if n % 2 == 0: + return False + + sqrt_n = int(math.floor(math.sqrt(n))) + for i in range(3, sqrt_n + 1, 2): + if n % i == 0: + return False + return True + + def main(): + with futures.ProcessPoolExecutor() as executor: + for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)): + print('%d is prime: %s' % (number, prime)) + + if __name__ == '__main__': + main() + +Future Objects +-------------- + +The :class:`Future` class encapulates the asynchronous execution of a callable. +:class:`Future` instances are created by :meth:`Executor.submit`. + +.. method:: Future.cancel() + + Attempt to cancel the call. If the call is currently being executed then + it cannot be cancelled and the method will return `False`, otherwise the call + will be cancelled and the method will return `True`. + +.. method:: Future.cancelled() + + Return `True` if the call was successfully cancelled. + +.. method:: Future.running() + + Return `True` if the call is currently being executed and cannot be + cancelled. + +.. method:: Future.done() + + Return `True` if the call was successfully cancelled or finished running. + +.. method:: Future.result(timeout=None) + + Return the value returned by the call. If the call hasn't yet completed then + this method will wait up to *timeout* seconds. If the call hasn't completed + in *timeout* seconds then a :exc:`TimeoutError` will be raised. *timeout* can + be an int or float.If *timeout* is not specified or ``None`` then there is no + limit to the wait time. + + If the future is cancelled before completing then :exc:`CancelledError` will + be raised. + + If the call raised then this method will raise the same exception. + +.. method:: Future.exception(timeout=None) + + Return the exception raised by the call. If the call hasn't yet completed + then this method will wait up to *timeout* seconds. If the call hasn't + completed in *timeout* seconds then a :exc:`TimeoutError` will be raised. + *timeout* can be an int or float. If *timeout* is not specified or ``None`` + then there is no limit to the wait time. + + If the future is cancelled before completing then :exc:`CancelledError` will + be raised. + + If the call completed without raising then ``None`` is returned. + +.. method:: Future.add_done_callback(fn) + + Attaches the callable *fn* to the future. *fn* will be called, with the + future as its only argument, when the future is cancelled or finishes + running. + + Added callables are called in the order that they were added and are always + called in a thread belonging to the process that added them. If the callable + raises an :exc:`Exception` then it will be logged and ignored. If the + callable raises another :exc:`BaseException` then the behavior is not + defined. + + If the future has already completed or been cancelled then *fn* will be + called immediately. + +Internal Future Methods +^^^^^^^^^^^^^^^^^^^^^^^ + +The following :class:`Future` methods are meant for use in unit tests and +:class:`Executor` implementations. + +.. method:: Future.set_running_or_notify_cancel() + + This method should only be called by :class:`Executor` implementations before + executing the work associated with the :class:`Future` and by unit tests. + + If the method returns `False` then the :class:`Future` was cancelled i.e. + :meth:`Future.cancel` was called and returned `True`. Any threads waiting + on the :class:`Future` completing (i.e. through :func:`as_completed` or + :func:`wait`) will be woken up. + + If the method returns `True` then the :class:`Future` was not cancelled + and has been put in the running state i.e. calls to + :meth:`Future.running` will return `True`. + + This method can only be called once and cannot be called after + :meth:`Future.set_result` or :meth:`Future.set_exception` have been + called. + +.. method:: Future.set_result(result) + + Sets the result of the work associated with the :class:`Future` to *result*. + + This method should only be used by Executor implementations and unit tests. + +.. method:: Future.set_exception(exception) + + Sets the result of the work associated with the :class:`Future` to the + :class:`Exception` *exception*. + + This method should only be used by Executor implementations and unit tests. + +Module Functions +---------------- + +.. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED) + + Wait for the :class:`Future` instances (possibly created by different + :class:`Executor` instances) given by *fs* to complete. Returns a named + 2-tuple of sets. The first set, named "done", contains the futures that + completed (finished or were cancelled) before the wait completed. The second + set, named "not_done", contains uncompleted futures. + + *timeout* can be used to control the maximum number of seconds to wait before + returning. *timeout* can be an int or float. If *timeout* is not specified or + ``None`` then there is no limit to the wait time. + + *return_when* indicates when this function should return. It must be one of + the following constants: + + +-----------------------------+----------------------------------------+ + | Constant | Description | + +=============================+========================================+ + | :const:`FIRST_COMPLETED` | The function will return when any | + | | future finishes or is cancelled. | + +-----------------------------+----------------------------------------+ + | :const:`FIRST_EXCEPTION` | The function will return when any | + | | future finishes by raising an | + | | exception. If no future raises an | + | | exception then it is equivalent to | + | | `ALL_COMPLETED`. | + +-----------------------------+----------------------------------------+ + | :const:`ALL_COMPLETED` | The function will return when all | + | | futures finish or are cancelled. | + +-----------------------------+----------------------------------------+ + +.. function:: as_completed(fs, timeout=None) + + Returns an iterator over the :class:`Future` instances (possibly created + by different :class:`Executor` instances) given by *fs* that yields futures + as they complete (finished or were cancelled). Any futures that completed + before :func:`as_completed()` was called will be yielded first. The returned + iterator raises a :exc:`TimeoutError` if :meth:`__next__()` is called and + the result isn't available after *timeout* seconds from the original call + to :func:`as_completed()`. *timeout* can be an int or float. If *timeout* + is not specified or ``None`` then there is no limit to the wait time. diff --git a/third_party/pythonfutures/docs/make.bat b/third_party/pythonfutures/docs/make.bat new file mode 100755 index 00000000..3e8021b5 --- /dev/null +++ b/third_party/pythonfutures/docs/make.bat @@ -0,0 +1,112 @@ +@ECHO OFF + +REM Command file for Sphinx documentation + +set SPHINXBUILD=sphinx-build +set ALLSPHINXOPTS=-d _build/doctrees %SPHINXOPTS% . +if NOT "%PAPER%" == "" ( + set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% +) + +if "%1" == "" goto help + +if "%1" == "help" ( + :help + echo.Please use `make ^` where ^ is one of + echo. html to make standalone HTML files + echo. dirhtml to make HTML files named index.html in directories + echo. pickle to make pickle files + echo. json to make JSON files + echo. htmlhelp to make HTML files and a HTML help project + echo. qthelp to make HTML files and a qthelp project + echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter + echo. changes to make an overview over all changed/added/deprecated items + echo. linkcheck to check all external links for integrity + echo. doctest to run all doctests embedded in the documentation if enabled + goto end +) + +if "%1" == "clean" ( + for /d %%i in (_build\*) do rmdir /q /s %%i + del /q /s _build\* + goto end +) + +if "%1" == "html" ( + %SPHINXBUILD% -b html %ALLSPHINXOPTS% _build/html + echo. + echo.Build finished. The HTML pages are in _build/html. + goto end +) + +if "%1" == "dirhtml" ( + %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% _build/dirhtml + echo. + echo.Build finished. The HTML pages are in _build/dirhtml. + goto end +) + +if "%1" == "pickle" ( + %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% _build/pickle + echo. + echo.Build finished; now you can process the pickle files. + goto end +) + +if "%1" == "json" ( + %SPHINXBUILD% -b json %ALLSPHINXOPTS% _build/json + echo. + echo.Build finished; now you can process the JSON files. + goto end +) + +if "%1" == "htmlhelp" ( + %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% _build/htmlhelp + echo. + echo.Build finished; now you can run HTML Help Workshop with the ^ +.hhp project file in _build/htmlhelp. + goto end +) + +if "%1" == "qthelp" ( + %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% _build/qthelp + echo. + echo.Build finished; now you can run "qcollectiongenerator" with the ^ +.qhcp project file in _build/qthelp, like this: + echo.^> qcollectiongenerator _build\qthelp\futures.qhcp + echo.To view the help file: + echo.^> assistant -collectionFile _build\qthelp\futures.ghc + goto end +) + +if "%1" == "latex" ( + %SPHINXBUILD% -b latex %ALLSPHINXOPTS% _build/latex + echo. + echo.Build finished; the LaTeX files are in _build/latex. + goto end +) + +if "%1" == "changes" ( + %SPHINXBUILD% -b changes %ALLSPHINXOPTS% _build/changes + echo. + echo.The overview file is in _build/changes. + goto end +) + +if "%1" == "linkcheck" ( + %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% _build/linkcheck + echo. + echo.Link check complete; look for any errors in the above output ^ +or in _build/linkcheck/output.txt. + goto end +) + +if "%1" == "doctest" ( + %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% _build/doctest + echo. + echo.Testing of doctests in the sources finished, look at the ^ +results in _build/doctest/output.txt. + goto end +) + +:end diff --git a/third_party/pythonfutures/futures/__init__.py b/third_party/pythonfutures/futures/__init__.py new file mode 100755 index 00000000..8f8b2348 --- /dev/null +++ b/third_party/pythonfutures/futures/__init__.py @@ -0,0 +1,24 @@ +# Copyright 2009 Brian Quinlan. All Rights Reserved. +# Licensed to PSF under a Contributor Agreement. + +"""Execute computations asynchronously using threads or processes.""" + +import warnings + +from concurrent.futures import (FIRST_COMPLETED, + FIRST_EXCEPTION, + ALL_COMPLETED, + CancelledError, + TimeoutError, + Future, + Executor, + wait, + as_completed, + ProcessPoolExecutor, + ThreadPoolExecutor) + +__author__ = 'Brian Quinlan (brian@sweetapp.com)' + +warnings.warn('The futures package has been deprecated. ' + 'Use the concurrent.futures package instead.', + DeprecationWarning) diff --git a/third_party/pythonfutures/futures/process.py b/third_party/pythonfutures/futures/process.py new file mode 100755 index 00000000..e9d37b16 --- /dev/null +++ b/third_party/pythonfutures/futures/process.py @@ -0,0 +1 @@ +from concurrent.futures import ProcessPoolExecutor diff --git a/third_party/pythonfutures/futures/thread.py b/third_party/pythonfutures/futures/thread.py new file mode 100755 index 00000000..f6bd05de --- /dev/null +++ b/third_party/pythonfutures/futures/thread.py @@ -0,0 +1 @@ +from concurrent.futures import ThreadPoolExecutor diff --git a/third_party/pythonfutures/primes.py b/third_party/pythonfutures/primes.py new file mode 100755 index 00000000..0da2b3e6 --- /dev/null +++ b/third_party/pythonfutures/primes.py @@ -0,0 +1,50 @@ +from __future__ import with_statement +import math +import time +import sys + +from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor + +PRIMES = [ + 112272535095293, + 112582705942171, + 112272535095293, + 115280095190773, + 115797848077099, + 117450548693743, + 993960000099397] + +def is_prime(n): + if n % 2 == 0: + return False + + sqrt_n = int(math.floor(math.sqrt(n))) + for i in range(3, sqrt_n + 1, 2): + if n % i == 0: + return False + return True + +def sequential(): + return list(map(is_prime, PRIMES)) + +def with_process_pool_executor(): + with ProcessPoolExecutor(10) as executor: + return list(executor.map(is_prime, PRIMES)) + +def with_thread_pool_executor(): + with ThreadPoolExecutor(10) as executor: + return list(executor.map(is_prime, PRIMES)) + +def main(): + for name, fn in [('sequential', sequential), + ('processes', with_process_pool_executor), + ('threads', with_thread_pool_executor)]: + sys.stdout.write('%s: ' % name.ljust(12)) + start = time.time() + if fn() != [True] * len(PRIMES): + sys.stdout.write('failed\n') + else: + sys.stdout.write('%.2f seconds\n' % (time.time() - start)) + +if __name__ == '__main__': + main() diff --git a/third_party/pythonfutures/setup.cfg b/third_party/pythonfutures/setup.cfg new file mode 100755 index 00000000..0a9f4f52 --- /dev/null +++ b/third_party/pythonfutures/setup.cfg @@ -0,0 +1,6 @@ +[build_sphinx] +source-dir = docs +build-dir = build/sphinx + +[upload_docs] +upload-dir = build/sphinx/html diff --git a/third_party/pythonfutures/setup.py b/third_party/pythonfutures/setup.py new file mode 100755 index 00000000..c08461ed --- /dev/null +++ b/third_party/pythonfutures/setup.py @@ -0,0 +1,33 @@ +#!/usr/bin/env python +import sys + +extras = {} +try: + from setuptools import setup + extras['zip_safe'] = False + if sys.version_info < (2, 6): + extras['install_requires'] = ['multiprocessing'] +except ImportError: + from distutils.core import setup + +setup(name='futures', + version='2.1.4', + description='Backport of the concurrent.futures package from Python 3.2', + author='Brian Quinlan', + author_email='brian@sweetapp.com', + maintainer='Alex Gronholm', + maintainer_email='alex.gronholm+pypi@nextday.fi', + url='http://code.google.com/p/pythonfutures', + download_url='http://pypi.python.org/pypi/futures/', + packages=['futures', 'concurrent', 'concurrent.futures'], + license='BSD', + classifiers=['License :: OSI Approved :: BSD License', + 'Development Status :: 5 - Production/Stable', + 'Intended Audience :: Developers', + 'Programming Language :: Python :: 2.5', + 'Programming Language :: Python :: 2.6', + 'Programming Language :: Python :: 2.7', + 'Programming Language :: Python :: 3', + 'Programming Language :: Python :: 3.1'], + **extras + ) diff --git a/third_party/pythonfutures/test_futures.py b/third_party/pythonfutures/test_futures.py new file mode 100755 index 00000000..dd7fd3e6 --- /dev/null +++ b/third_party/pythonfutures/test_futures.py @@ -0,0 +1,723 @@ +from __future__ import with_statement +import os +import subprocess +import sys +import threading +import functools +import contextlib +import logging +import re +import time + +from concurrent import futures +from concurrent.futures._base import ( + PENDING, RUNNING, CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED, Future) + +try: + import unittest2 as unittest +except ImportError: + import unittest + +try: + from StringIO import StringIO +except ImportError: + from io import StringIO + +try: + from test import test_support +except ImportError: + from test import support as test_support + +try: + next +except NameError: + next = lambda x: x.next() + + +def reap_threads(func): + """Use this function when threads are being used. This will + ensure that the threads are cleaned up even when the test fails. + If threading is unavailable this function does nothing. + """ + @functools.wraps(func) + def decorator(*args): + key = test_support.threading_setup() + try: + return func(*args) + finally: + test_support.threading_cleanup(*key) + return decorator + + +# Executing the interpreter in a subprocess +def _assert_python(expected_success, *args, **env_vars): + cmd_line = [sys.executable] + if not env_vars: + cmd_line.append('-E') + # Need to preserve the original environment, for in-place testing of + # shared library builds. + env = os.environ.copy() + # But a special flag that can be set to override -- in this case, the + # caller is responsible to pass the full environment. + if env_vars.pop('__cleanenv', None): + env = {} + env.update(env_vars) + cmd_line.extend(args) + p = subprocess.Popen(cmd_line, stdin=subprocess.PIPE, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + env=env) + try: + out, err = p.communicate() + finally: + subprocess._cleanup() + p.stdout.close() + p.stderr.close() + rc = p.returncode + err = strip_python_stderr(err) + if (rc and expected_success) or (not rc and not expected_success): + raise AssertionError( + "Process return code is %d, " + "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore'))) + return rc, out, err + + +def assert_python_ok(*args, **env_vars): + """ + Assert that running the interpreter with `args` and optional environment + variables `env_vars` is ok and return a (return code, stdout, stderr) tuple. + """ + return _assert_python(True, *args, **env_vars) + + +def strip_python_stderr(stderr): + """Strip the stderr of a Python process from potential debug output + emitted by the interpreter. + + This will typically be run on the result of the communicate() method + of a subprocess.Popen object. + """ + stderr = re.sub(r"\[\d+ refs\]\r?\n?$".encode(), "".encode(), stderr).strip() + return stderr + + +@contextlib.contextmanager +def captured_stderr(): + """Return a context manager used by captured_stdout/stdin/stderr + that temporarily replaces the sys stream *stream_name* with a StringIO.""" + logging_stream = StringIO() + handler = logging.StreamHandler(logging_stream) + logging.root.addHandler(handler) + + try: + yield logging_stream + finally: + logging.root.removeHandler(handler) + + +def create_future(state=PENDING, exception=None, result=None): + f = Future() + f._state = state + f._exception = exception + f._result = result + return f + + +PENDING_FUTURE = create_future(state=PENDING) +RUNNING_FUTURE = create_future(state=RUNNING) +CANCELLED_FUTURE = create_future(state=CANCELLED) +CANCELLED_AND_NOTIFIED_FUTURE = create_future(state=CANCELLED_AND_NOTIFIED) +EXCEPTION_FUTURE = create_future(state=FINISHED, exception=IOError()) +SUCCESSFUL_FUTURE = create_future(state=FINISHED, result=42) + + +def mul(x, y): + return x * y + + +def sleep_and_raise(t): + time.sleep(t) + raise Exception('this is an exception') + +def sleep_and_print(t, msg): + time.sleep(t) + print(msg) + sys.stdout.flush() + + +class ExecutorMixin: + worker_count = 5 + + def setUp(self): + self.t1 = time.time() + try: + self.executor = self.executor_type(max_workers=self.worker_count) + except NotImplementedError: + e = sys.exc_info()[1] + self.skipTest(str(e)) + self._prime_executor() + + def tearDown(self): + self.executor.shutdown(wait=True) + dt = time.time() - self.t1 + if test_support.verbose: + print("%.2fs" % dt) + self.assertLess(dt, 60, "synchronization issue: test lasted too long") + + def _prime_executor(self): + # Make sure that the executor is ready to do work before running the + # tests. This should reduce the probability of timeouts in the tests. + futures = [self.executor.submit(time.sleep, 0.1) + for _ in range(self.worker_count)] + + for f in futures: + f.result() + + +class ThreadPoolMixin(ExecutorMixin): + executor_type = futures.ThreadPoolExecutor + + +class ProcessPoolMixin(ExecutorMixin): + executor_type = futures.ProcessPoolExecutor + + +class ExecutorShutdownTest(unittest.TestCase): + def test_run_after_shutdown(self): + self.executor.shutdown() + self.assertRaises(RuntimeError, + self.executor.submit, + pow, 2, 5) + + def test_interpreter_shutdown(self): + # Test the atexit hook for shutdown of worker threads and processes + rc, out, err = assert_python_ok('-c', """if 1: + from concurrent.futures import %s + from time import sleep + from test_futures import sleep_and_print + t = %s(5) + t.submit(sleep_and_print, 1.0, "apple") + """ % (self.executor_type.__name__, self.executor_type.__name__)) + # Errors in atexit hooks don't change the process exit code, check + # stderr manually. + self.assertFalse(err) + self.assertEqual(out.strip(), "apple".encode()) + + def test_hang_issue12364(self): + fs = [self.executor.submit(time.sleep, 0.1) for _ in range(50)] + self.executor.shutdown() + for f in fs: + f.result() + + +class ThreadPoolShutdownTest(ThreadPoolMixin, ExecutorShutdownTest): + def _prime_executor(self): + pass + + def test_threads_terminate(self): + self.executor.submit(mul, 21, 2) + self.executor.submit(mul, 6, 7) + self.executor.submit(mul, 3, 14) + self.assertEqual(len(self.executor._threads), 3) + self.executor.shutdown() + for t in self.executor._threads: + t.join() + + def test_context_manager_shutdown(self): + with futures.ThreadPoolExecutor(max_workers=5) as e: + executor = e + self.assertEqual(list(e.map(abs, range(-5, 5))), + [5, 4, 3, 2, 1, 0, 1, 2, 3, 4]) + + for t in executor._threads: + t.join() + + def test_del_shutdown(self): + executor = futures.ThreadPoolExecutor(max_workers=5) + executor.map(abs, range(-5, 5)) + threads = executor._threads + del executor + + for t in threads: + t.join() + + +class ProcessPoolShutdownTest(ProcessPoolMixin, ExecutorShutdownTest): + def _prime_executor(self): + pass + + def test_processes_terminate(self): + self.executor.submit(mul, 21, 2) + self.executor.submit(mul, 6, 7) + self.executor.submit(mul, 3, 14) + self.assertEqual(len(self.executor._processes), 5) + processes = self.executor._processes + self.executor.shutdown() + + for p in processes: + p.join() + + def test_context_manager_shutdown(self): + with futures.ProcessPoolExecutor(max_workers=5) as e: + processes = e._processes + self.assertEqual(list(e.map(abs, range(-5, 5))), + [5, 4, 3, 2, 1, 0, 1, 2, 3, 4]) + + for p in processes: + p.join() + + def test_del_shutdown(self): + executor = futures.ProcessPoolExecutor(max_workers=5) + list(executor.map(abs, range(-5, 5))) + queue_management_thread = executor._queue_management_thread + processes = executor._processes + del executor + + queue_management_thread.join() + for p in processes: + p.join() + + +class WaitTests(unittest.TestCase): + + def test_first_completed(self): + future1 = self.executor.submit(mul, 21, 2) + future2 = self.executor.submit(time.sleep, 1.5) + + done, not_done = futures.wait( + [CANCELLED_FUTURE, future1, future2], + return_when=futures.FIRST_COMPLETED) + + self.assertEqual(set([future1]), done) + self.assertEqual(set([CANCELLED_FUTURE, future2]), not_done) + + def test_first_completed_some_already_completed(self): + future1 = self.executor.submit(time.sleep, 1.5) + + finished, pending = futures.wait( + [CANCELLED_AND_NOTIFIED_FUTURE, SUCCESSFUL_FUTURE, future1], + return_when=futures.FIRST_COMPLETED) + + self.assertEqual( + set([CANCELLED_AND_NOTIFIED_FUTURE, SUCCESSFUL_FUTURE]), + finished) + self.assertEqual(set([future1]), pending) + + def test_first_exception(self): + future1 = self.executor.submit(mul, 2, 21) + future2 = self.executor.submit(sleep_and_raise, 1.5) + future3 = self.executor.submit(time.sleep, 3) + + finished, pending = futures.wait( + [future1, future2, future3], + return_when=futures.FIRST_EXCEPTION) + + self.assertEqual(set([future1, future2]), finished) + self.assertEqual(set([future3]), pending) + + def test_first_exception_some_already_complete(self): + future1 = self.executor.submit(divmod, 21, 0) + future2 = self.executor.submit(time.sleep, 1.5) + + finished, pending = futures.wait( + [SUCCESSFUL_FUTURE, + CANCELLED_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, + future1, future2], + return_when=futures.FIRST_EXCEPTION) + + self.assertEqual(set([SUCCESSFUL_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, + future1]), finished) + self.assertEqual(set([CANCELLED_FUTURE, future2]), pending) + + def test_first_exception_one_already_failed(self): + future1 = self.executor.submit(time.sleep, 2) + + finished, pending = futures.wait( + [EXCEPTION_FUTURE, future1], + return_when=futures.FIRST_EXCEPTION) + + self.assertEqual(set([EXCEPTION_FUTURE]), finished) + self.assertEqual(set([future1]), pending) + + def test_all_completed(self): + future1 = self.executor.submit(divmod, 2, 0) + future2 = self.executor.submit(mul, 2, 21) + + finished, pending = futures.wait( + [SUCCESSFUL_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + future1, + future2], + return_when=futures.ALL_COMPLETED) + + self.assertEqual(set([SUCCESSFUL_FUTURE, + CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + future1, + future2]), finished) + self.assertEqual(set(), pending) + + def test_timeout(self): + future1 = self.executor.submit(mul, 6, 7) + future2 = self.executor.submit(time.sleep, 3) + + finished, pending = futures.wait( + [CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE, + future1, future2], + timeout=1.5, + return_when=futures.ALL_COMPLETED) + + self.assertEqual(set([CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE, + future1]), finished) + self.assertEqual(set([future2]), pending) + + +class ThreadPoolWaitTests(ThreadPoolMixin, WaitTests): + + def test_pending_calls_race(self): + # Issue #14406: multi-threaded race condition when waiting on all + # futures. + event = threading.Event() + def future_func(): + event.wait() + oldswitchinterval = sys.getcheckinterval() + sys.setcheckinterval(1) + try: + fs = set(self.executor.submit(future_func) for i in range(100)) + event.set() + futures.wait(fs, return_when=futures.ALL_COMPLETED) + finally: + sys.setcheckinterval(oldswitchinterval) + + +class ProcessPoolWaitTests(ProcessPoolMixin, WaitTests): + pass + + +class AsCompletedTests(unittest.TestCase): + # TODO(brian@sweetapp.com): Should have a test with a non-zero timeout. + def test_no_timeout(self): + future1 = self.executor.submit(mul, 2, 21) + future2 = self.executor.submit(mul, 7, 6) + + completed = set(futures.as_completed( + [CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE, + future1, future2])) + self.assertEqual(set( + [CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE, + future1, future2]), + completed) + + def test_zero_timeout(self): + future1 = self.executor.submit(time.sleep, 2) + completed_futures = set() + try: + for future in futures.as_completed( + [CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE, + future1], + timeout=0): + completed_futures.add(future) + except futures.TimeoutError: + pass + + self.assertEqual(set([CANCELLED_AND_NOTIFIED_FUTURE, + EXCEPTION_FUTURE, + SUCCESSFUL_FUTURE]), + completed_futures) + + +class ThreadPoolAsCompletedTests(ThreadPoolMixin, AsCompletedTests): + pass + + +class ProcessPoolAsCompletedTests(ProcessPoolMixin, AsCompletedTests): + pass + + +class ExecutorTest(unittest.TestCase): + # Executor.shutdown() and context manager usage is tested by + # ExecutorShutdownTest. + def test_submit(self): + future = self.executor.submit(pow, 2, 8) + self.assertEqual(256, future.result()) + + def test_submit_keyword(self): + future = self.executor.submit(mul, 2, y=8) + self.assertEqual(16, future.result()) + + def test_map(self): + self.assertEqual( + list(self.executor.map(pow, range(10), range(10))), + list(map(pow, range(10), range(10)))) + + def test_map_exception(self): + i = self.executor.map(divmod, [1, 1, 1, 1], [2, 3, 0, 5]) + self.assertEqual(next(i), (0, 1)) + self.assertEqual(next(i), (0, 1)) + self.assertRaises(ZeroDivisionError, next, i) + + def test_map_timeout(self): + results = [] + try: + for i in self.executor.map(time.sleep, + [0, 0, 3], + timeout=1.5): + results.append(i) + except futures.TimeoutError: + pass + else: + self.fail('expected TimeoutError') + + self.assertEqual([None, None], results) + + +class ThreadPoolExecutorTest(ThreadPoolMixin, ExecutorTest): + pass + + +class ProcessPoolExecutorTest(ProcessPoolMixin, ExecutorTest): + pass + + +class FutureTests(unittest.TestCase): + def test_done_callback_with_result(self): + callback_result = [None] + def fn(callback_future): + callback_result[0] = callback_future.result() + + f = Future() + f.add_done_callback(fn) + f.set_result(5) + self.assertEqual(5, callback_result[0]) + + def test_done_callback_with_exception(self): + callback_exception = [None] + def fn(callback_future): + callback_exception[0] = callback_future.exception() + + f = Future() + f.add_done_callback(fn) + f.set_exception(Exception('test')) + self.assertEqual(('test',), callback_exception[0].args) + + def test_done_callback_with_cancel(self): + was_cancelled = [None] + def fn(callback_future): + was_cancelled[0] = callback_future.cancelled() + + f = Future() + f.add_done_callback(fn) + self.assertTrue(f.cancel()) + self.assertTrue(was_cancelled[0]) + + def test_done_callback_raises(self): + with captured_stderr() as stderr: + raising_was_called = [False] + fn_was_called = [False] + + def raising_fn(callback_future): + raising_was_called[0] = True + raise Exception('doh!') + + def fn(callback_future): + fn_was_called[0] = True + + f = Future() + f.add_done_callback(raising_fn) + f.add_done_callback(fn) + f.set_result(5) + self.assertTrue(raising_was_called) + self.assertTrue(fn_was_called) + self.assertIn('Exception: doh!', stderr.getvalue()) + + def test_done_callback_already_successful(self): + callback_result = [None] + def fn(callback_future): + callback_result[0] = callback_future.result() + + f = Future() + f.set_result(5) + f.add_done_callback(fn) + self.assertEqual(5, callback_result[0]) + + def test_done_callback_already_failed(self): + callback_exception = [None] + def fn(callback_future): + callback_exception[0] = callback_future.exception() + + f = Future() + f.set_exception(Exception('test')) + f.add_done_callback(fn) + self.assertEqual(('test',), callback_exception[0].args) + + def test_done_callback_already_cancelled(self): + was_cancelled = [None] + def fn(callback_future): + was_cancelled[0] = callback_future.cancelled() + + f = Future() + self.assertTrue(f.cancel()) + f.add_done_callback(fn) + self.assertTrue(was_cancelled[0]) + + def test_repr(self): + self.assertRegexpMatches(repr(PENDING_FUTURE), + '') + self.assertRegexpMatches(repr(RUNNING_FUTURE), + '') + self.assertRegexpMatches(repr(CANCELLED_FUTURE), + '') + self.assertRegexpMatches(repr(CANCELLED_AND_NOTIFIED_FUTURE), + '') + self.assertRegexpMatches( + repr(EXCEPTION_FUTURE), + '') + self.assertRegexpMatches( + repr(SUCCESSFUL_FUTURE), + '') + + def test_cancel(self): + f1 = create_future(state=PENDING) + f2 = create_future(state=RUNNING) + f3 = create_future(state=CANCELLED) + f4 = create_future(state=CANCELLED_AND_NOTIFIED) + f5 = create_future(state=FINISHED, exception=IOError()) + f6 = create_future(state=FINISHED, result=5) + + self.assertTrue(f1.cancel()) + self.assertEqual(f1._state, CANCELLED) + + self.assertFalse(f2.cancel()) + self.assertEqual(f2._state, RUNNING) + + self.assertTrue(f3.cancel()) + self.assertEqual(f3._state, CANCELLED) + + self.assertTrue(f4.cancel()) + self.assertEqual(f4._state, CANCELLED_AND_NOTIFIED) + + self.assertFalse(f5.cancel()) + self.assertEqual(f5._state, FINISHED) + + self.assertFalse(f6.cancel()) + self.assertEqual(f6._state, FINISHED) + + def test_cancelled(self): + self.assertFalse(PENDING_FUTURE.cancelled()) + self.assertFalse(RUNNING_FUTURE.cancelled()) + self.assertTrue(CANCELLED_FUTURE.cancelled()) + self.assertTrue(CANCELLED_AND_NOTIFIED_FUTURE.cancelled()) + self.assertFalse(EXCEPTION_FUTURE.cancelled()) + self.assertFalse(SUCCESSFUL_FUTURE.cancelled()) + + def test_done(self): + self.assertFalse(PENDING_FUTURE.done()) + self.assertFalse(RUNNING_FUTURE.done()) + self.assertTrue(CANCELLED_FUTURE.done()) + self.assertTrue(CANCELLED_AND_NOTIFIED_FUTURE.done()) + self.assertTrue(EXCEPTION_FUTURE.done()) + self.assertTrue(SUCCESSFUL_FUTURE.done()) + + def test_running(self): + self.assertFalse(PENDING_FUTURE.running()) + self.assertTrue(RUNNING_FUTURE.running()) + self.assertFalse(CANCELLED_FUTURE.running()) + self.assertFalse(CANCELLED_AND_NOTIFIED_FUTURE.running()) + self.assertFalse(EXCEPTION_FUTURE.running()) + self.assertFalse(SUCCESSFUL_FUTURE.running()) + + def test_result_with_timeout(self): + self.assertRaises(futures.TimeoutError, + PENDING_FUTURE.result, timeout=0) + self.assertRaises(futures.TimeoutError, + RUNNING_FUTURE.result, timeout=0) + self.assertRaises(futures.CancelledError, + CANCELLED_FUTURE.result, timeout=0) + self.assertRaises(futures.CancelledError, + CANCELLED_AND_NOTIFIED_FUTURE.result, timeout=0) + self.assertRaises(IOError, EXCEPTION_FUTURE.result, timeout=0) + self.assertEqual(SUCCESSFUL_FUTURE.result(timeout=0), 42) + + def test_result_with_success(self): + # TODO(brian@sweetapp.com): This test is timing dependant. + def notification(): + # Wait until the main thread is waiting for the result. + time.sleep(1) + f1.set_result(42) + + f1 = create_future(state=PENDING) + t = threading.Thread(target=notification) + t.start() + + self.assertEqual(f1.result(timeout=5), 42) + + def test_result_with_cancel(self): + # TODO(brian@sweetapp.com): This test is timing dependant. + def notification(): + # Wait until the main thread is waiting for the result. + time.sleep(1) + f1.cancel() + + f1 = create_future(state=PENDING) + t = threading.Thread(target=notification) + t.start() + + self.assertRaises(futures.CancelledError, f1.result, timeout=5) + + def test_exception_with_timeout(self): + self.assertRaises(futures.TimeoutError, + PENDING_FUTURE.exception, timeout=0) + self.assertRaises(futures.TimeoutError, + RUNNING_FUTURE.exception, timeout=0) + self.assertRaises(futures.CancelledError, + CANCELLED_FUTURE.exception, timeout=0) + self.assertRaises(futures.CancelledError, + CANCELLED_AND_NOTIFIED_FUTURE.exception, timeout=0) + self.assertTrue(isinstance(EXCEPTION_FUTURE.exception(timeout=0), + IOError)) + self.assertEqual(SUCCESSFUL_FUTURE.exception(timeout=0), None) + + def test_exception_with_success(self): + def notification(): + # Wait until the main thread is waiting for the exception. + time.sleep(1) + with f1._condition: + f1._state = FINISHED + f1._exception = IOError() + f1._condition.notify_all() + + f1 = create_future(state=PENDING) + t = threading.Thread(target=notification) + t.start() + + self.assertTrue(isinstance(f1.exception(timeout=5), IOError)) + +@reap_threads +def test_main(): + try: + test_support.run_unittest(ProcessPoolExecutorTest, + ThreadPoolExecutorTest, + ProcessPoolWaitTests, + ThreadPoolWaitTests, + ProcessPoolAsCompletedTests, + ThreadPoolAsCompletedTests, + FutureTests, + ProcessPoolShutdownTest, + ThreadPoolShutdownTest) + finally: + test_support.reap_children() + +if __name__ == "__main__": + test_main() diff --git a/third_party/pythonfutures/tox.ini b/third_party/pythonfutures/tox.ini new file mode 100755 index 00000000..c1ff2f13 --- /dev/null +++ b/third_party/pythonfutures/tox.ini @@ -0,0 +1,8 @@ +[tox] +envlist = py26,py27,py31 + +[testenv] +commands={envpython} test_futures.py [] + +[testenv:py26] +deps=unittest2 diff --git a/third_party/requests b/third_party/requests new file mode 160000 index 00000000..9968a10f --- /dev/null +++ b/third_party/requests @@ -0,0 +1 @@ +Subproject commit 9968a10fcfad7268b552808c4f8946eecafc956a diff --git a/third_party/requests-futures b/third_party/requests-futures new file mode 160000 index 00000000..79e52563 --- /dev/null +++ b/third_party/requests-futures @@ -0,0 +1 @@ +Subproject commit 79e525634e33d86cb5f3e62dcba88f98fe4f4cbc diff --git a/third_party/retries/retries.py b/third_party/retries/retries.py new file mode 100644 index 00000000..1d7131d5 --- /dev/null +++ b/third_party/retries/retries.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python +# +# Copyright 2012 by Jeff Laughlin Consulting LLC +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + + +import sys +from time import sleep + +# Source: https://gist.github.com/n1ywb/2570004 + +def example_exc_handler(tries_remaining, exception, delay): + """Example exception handler; prints a warning to stderr. + + tries_remaining: The number of tries remaining. + exception: The exception instance which was raised. + """ + print >> sys.stderr, "Caught '%s', %d tries remaining, sleeping for %s seconds" % (exception, tries_remaining, delay) + + +def retries(max_tries, delay=1, backoff=2, exceptions=(Exception,), hook=None): + """Function decorator implementing retrying logic. + + delay: Sleep this many seconds * backoff * try number after failure + backoff: Multiply delay by this factor after each failure + exceptions: A tuple of exception classes; default (Exception,) + hook: A function with the signature myhook(tries_remaining, exception); + default None + + The decorator will call the function up to max_tries times if it raises + an exception. + + By default it catches instances of the Exception class and subclasses. + This will recover after all but the most fatal errors. You may specify a + custom tuple of exception classes with the 'exceptions' argument; the + function will only be retried if it raises one of the specified + exceptions. + + Additionally you may specify a hook function which will be called prior + to retrying with the number of remaining tries and the exception instance; + see given example. This is primarily intended to give the opportunity to + log the failure. Hook is not called after failure if no retries remain. + """ + def dec(func): + def f2(*args, **kwargs): + mydelay = delay + tries = range(max_tries) + tries.reverse() + for tries_remaining in tries: + try: + return func(*args, **kwargs) + except exceptions as e: + if tries_remaining > 0: + if hook is not None: + hook(tries_remaining, e, mydelay) + sleep(mydelay) + mydelay = mydelay * backoff + else: + raise + else: + break + return f2 + return dec diff --git a/third_party/waitress b/third_party/waitress new file mode 160000 index 00000000..c957f1d7 --- /dev/null +++ b/third_party/waitress @@ -0,0 +1 @@ +Subproject commit c957f1d70ab82ba15ce0dce8fca7cf70fe2f2b97