[READY] Import the requests module lazily
The requests module is slow to import. See https://github.com/kennethreitz/requests/issues/3213. We should lazy load it to improve startup time. We do that by adding two methods to the `BaseRequest` class: one that returns the requests module and another the session object since it depends on the `request-futures` module and `requests-futures` imports `requests`. In addition, we make sure that no requests are sent at startup otherwise there would be no point to lazy load these. These requests would fail anyway since the server can't be ready yet.
Here are the improvements on startup time:
<table>
<tr>
<th rowspan="2">Platform</th>
<th colspan="2">First run (ms)</th>
<th colspan="2">Subsequent runs (ms)</th>
</tr>
<tr>
<td>Before</td>
<td>After</td>
<td>Before</td>
<td>After</td>
</tr>
<tr>
<td>Ubuntu 16.04 64-bit</td>
<td>240</td>
<td>131</td>
<td>173</td>
<td>74</td>
</tr>
<tr>
<td>macOS 10.12</td>
<td>435</td>
<td>315</td>
<td>261</td>
<td>208</td>
</tr>
<tr>
<td>Windows 10 64-bit</td>
<td>894</td>
<td>594</td>
<td>359</td>
<td>247</td>
</tr>
</table>
*Results obtained by running the `prof.py` script from [this branch](https://github.com/micbou/YouCompleteMe/tree/profiling-startup). The difference between first run and subsequent runs is Python bytecode generation (`*.pyc` files).*
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2563)
<!-- Reviewable:end -->
YCM should start the same whether Vim is running in GUI or not. Otherwise, this
makes it difficult for other plugins to interact with YCM at startup because
they can't predict how YCM will start unless they also check if Vim is running
in GUI which is unreasonable. Since we can't load it at Vim startup because of
a deadlock issue, we should always defer it at the VimEnter event.
[READY] Refactor diagnostic commands
This PR moves the logic of the `YcmDiags` and `YcmForceCompileAndDiagnostics` commands to the Python layer. It replaces the `echo` calls by `PostVimMessage` ones to display message on the status line and it always prints `Diagnostics refreshed` once compilation is done. This has the benefit of clearing the message `Forcing compilation, this will block Vim until done.` and avoiding the `Press Enter or type command to continue` prompt.
This also adds the `YcmLocationOpened` autocommand which is the same as `YcmQuickFixOpened` but for the location list window.
Fixes#2543.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2548)
<!-- Reviewable:end -->
[RFC] Send requests again when server becomes ready
Since PR #2517, the `BufferVisit` and `FileReadyToParse` requests are ignored until the server is ready. We need to send them again when it becomes ready so that identifiers and snippets for the current buffer are properly parsed. This is done in the `s:OnFileReadyToParse` function since this function is regularly called. We also try to set the omnifunc since it depends on a response from the server and is only set when entering a buffer or changing its filetype. This replaces the code that consisted of setting the omnifunc when entering insert mode for the first time.
A cleaner solution would be to do that asynchronously but we can't with our Vim version requirement.
Fixes#2541.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2547)
<!-- Reviewable:end -->
Don't use VimEnter initialization except when stating gui
# PR Prelude
- [x] I have read and understood YCM's [CONTRIBUTING][cont] document.
- [x] I have read and understood YCM's [CODE_OF_CONDUCT][code] document.
- [x] I have included tests for the changes in my PR. If not, I have included a
rationale for why I haven't.
- [x] **I understand my PR may be closed if it becomes obvious I didn't
actually perform all of these steps.**
# Why this change is necessary and useful
This is another attempt at:
https://github.com/Valloric/YouCompleteMe/pull/2473
Which removed the apparently flawed attempt at "lazy" loading of YCM. While it fails to get the load out of the critical startup path, it *does* serve the useful purpose when starting up gvim of avoiding a deadlock situation in gvim.
So, this time, we keep VimEnter, but only for the gui-starting case. We update the comment to explain what is actually happening. And we can keep the docs about how to defer loading.
[cont]: https://github.com/Valloric/YouCompleteMe/blob/master/CONTRIBUTING.md
[code]: https://github.com/Valloric/YouCompleteMe/blob/master/CODE_OF_CONDUCT.md
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2475)
<!-- Reviewable:end -->
Updated ycmd to ec7a154 to fix racerd build issues
- [x] I have read and understood YCM's [CONTRIBUTING][cont] document.
- [x] I have read and understood YCM's [CODE_OF_CONDUCT][code] document.
- [x] I have included tests for the changes in my PR. If not, I have included a
rationale for why I haven't.
- [x] **I understand my PR may be closed if it becomes obvious I didn't
actually perform all of these steps.**
# Why this change is necessary and useful
Building racerd fails with the current submodule version of ycmd. This updates ycmd to the latest commit (ec7a154 at the time of writing) and fixes racerd failing to build. No additional tests were included because this is an update a library with its own tests.
Closes#2480.
[cont]: https://github.com/Valloric/YouCompleteMe/blob/master/CONTRIBUTING.md
[code]: https://github.com/Valloric/YouCompleteMe/blob/master/CODE_OF_CONDUCT.md
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2531)
<!-- Reviewable:end -->
Use bang version of `normal`
Fixes#2519
# PR Prelude
Thank you for working on YCM! :)
**Please complete these steps and check these boxes (by putting an `x` inside
the brackets) _before_ filing your PR:**
- [x] I have read and understood YCM's [CONTRIBUTING][cont] document.
- [x] I have read and understood YCM's [CODE_OF_CONDUCT][code] document.
- [x] I have included tests for the changes in my PR. If not, I have included a
rationale for why I haven't.
- [x] **I understand my PR may be closed if it becomes obvious I didn't
actually perform all of these steps.**
# Why this change is necessary and useful
[Please explain **in detail** why the changes in this PR are needed.]
[cont]: https://github.com/Valloric/YouCompleteMe/blob/master/CONTRIBUTING.md
[code]: https://github.com/Valloric/YouCompleteMe/blob/master/CODE_OF_CONDUCT.md
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2520)
<!-- Reviewable:end -->
[READY] Remove healthy check
This PR removes the code that, if the server is not yet healthy, tries in a separate thread to send the same request at different intervals until it gets a response or reaches 5 failed attempts.
Main issue with that code is that it blocks Vim if a synchronous request (`semantic_completion_available`, `run_completer_command`, etc.) is sent while the server is not up. See issue #2071.
Other issues are that it's hard to understand how this code behaves (it combines threads and retries), it becomes useless once the server is healthy (because `SERVER_HEALTHY` is cached and never reset), it depends on arbitrary parameters (why 5 retries with a delay of 0.5s multiplied by 1.5 at each attempt?), and it has a package dependency `retries`.
Now that we catch all server exceptions (PR #2453) and define a very short connection timeout (PR #2514), we can just drop this code.
This change gives a huge improvement in startup time on Windows (don't worry, more improvements are coming for all platforms). The reason is that `requests` (probably `urllib3`) tries for ~1s to check if the server is healthy on Windows while it's almost instantaneous on other platforms.
<table>
<tr>
<th rowspan="2">Platform</th>
<th colspan="2">First run (ms)</th>
<th colspan="2">Subsequent runs (ms)</th>
</tr>
<tr>
<td>Before</td>
<td>After</td>
<td>Before</td>
<td>After</td>
</tr>
<tr>
<td>Ubuntu 16.04 64-bit</td>
<td>228</td>
<td>248</td>
<td>176</td>
<td>151</td>
</tr>
<tr>
<td>macOS 10.12</td>
<td>433</td>
<td>417</td>
<td>263</td>
<td>263</td>
</tr>
<tr>
<td>Windows 10 64-bit</td>
<td>1861</td>
<td>844</td>
<td>814</td>
<td>292</td>
</tr>
</table>
These results were obtained by running the `prof.py` script from [this branch](https://github.com/micbou/YouCompleteMe/tree/profiling-startup). The difference between first run and subsequent runs is Python bytecode generation (`*.pyc` files).
Fixes#2071.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2517)
<!-- Reviewable:end -->
On non-Windows platforms, a ConnectionError exception is raised instead
of a ConnectTimeout one when the connection cannot be established (e.g.
the server crashed). Since the latter is a subclass of the former, we
only need to catch ConnectionError.
[READY] Rely on connect timeout instead of checking that the server is alive
Currently, we always check that the ycmd process is up (with the `IsServerAlive` method) before sending a request. Without this check, each request could block Vim until a `NewConnectionError` exception is raised if the server crashed. This is the case on Windows where it takes ~1s before the exception is raised which makes Vim unusable. However, even with this check, Vim may still be blocked in the following cases:
- the server crashes just after the check but before sending the request;
- the server is up but unresponsive (e.g. its port is closed).
To avoid both cases, we instead use [the connect timeout parameter from Requests](http://docs.python-requests.org/en/master/user/advanced/?highlight=connect%20timeout#timeouts) and set it to a duration sufficiently short (10 ms) that the blocking can't be noticed by the user. Since the server is supposed to run locally (this is what YCM is designed for), 10ms is largely enough to establish a connection.
The `IsServerAlive` check is removed almost everywhere except in `OnFileReadyToParse` because we still want to notify the user if the server crashed.
This change makes it possible to not have to [wait for the server to be healthy before sending asynchronous requests](https://github.com/Valloric/YouCompleteMe/blob/master/python/ycm/client/base_request.py#L137-L138). This will dramatically improve startup time (see issue #2085) and fixes#2071. Next PR once this one is merged.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2514)
<!-- Reviewable:end -->
[READY] Include subservers logfiles in YcmToggleLogs command
Thanks to PR #2342, we can easily update the `:YcmToggleLogs` command to also list the subservers logfiles and open them directly in Vim. For instance, when editing a Python file:
```
:YcmToggleLogs
Available logfiles are:
jedihttp_55438_stderr_g9rk18nc.log
jedihttp_55438_stdout_duc8kfqm.log
ycm_ttssiu.log
ycmd_55432_stderr_b9lnwf.log
ycmd_55432_stdout_jyakxs.log
```
Then
```
:YcmToggleLogs jedihttp_55438_stderr_g9rk18nc.log
```
to open the JediHTTP stderr logfile in Vim.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2506)
<!-- Reviewable:end -->
Disable sitecustomize when checking python version
# PR Prelude
Thank you for working on YCM! :)
**Please complete these steps and check these boxes (by putting an `x` inside
the brackets) _before_ filing your PR:**
- [X] I have read and understood YCM's [CONTRIBUTING][cont] document.
- [X] I have read and understood YCM's [CODE_OF_CONDUCT][code] document.
- [X] I have included tests for the changes in my PR. If not, I have included a
rationale for why I haven't:
- This seems quite difficult to test in a cross-platform, cross-version fashion, but
- I might have missed something, I am not a python expert. :)
- [X] **I understand my PR may be closed if it becomes obvious I didn't
actually perform all of these steps.**
# Why this change is necessary and useful
[Please explain **in detail** why the changes in this PR are needed.]
Sitecustomize files can be slow or broken and might hang editor
startup. Since they aren't necessary for checking the python version,
disable them to get a small speed boost for everyone, and an editor
that doesn't hang indefinitely on startup if the sitecustomize gets
into an infinite loop (due to a bad NFS mount or similar).
[cont]: https://github.com/Valloric/YouCompleteMe/blob/master/CONTRIBUTING.md
[code]: https://github.com/Valloric/YouCompleteMe/blob/master/CODE_OF_CONDUCT.md
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2502)
<!-- Reviewable:end -->
Sitecustomize files can be slow or broken and might hang editor
startup. Since they aren't necessary for checking the python version,
disable them to get a small speed boost for everyone, and an editor
that doesn't hang indefinitely on startup if the sitecustomize gets
into an infinite loop (due to a bad NFS mount or similar).
[READY] Update readme for compilation database support
# PR Prelude
Thank you for working on YCM! :)
**Please complete these steps and check these boxes (by putting an `x` inside
the brackets) _before_ filing your PR:**
- [X] I have read and understood YCM's [CONTRIBUTING][cont] document.
- [X] I have read and understood YCM's [CODE_OF_CONDUCT][code] document.
- [X] I have included tests for the changes in my PR. If not, I have included a
rationale for why I haven't.
> only changes docs
- [X] **I understand my PR may be closed if it becomes obvious I didn't
actually perform all of these steps.**
# Why this change is necessary and useful
This change:
- updates the c-family completer documentation to describe the built in support for compilation databases added in https://github.com/Valloric/ycmd/pull/680
- explains more about why ycmd needs compiler flags, and how to go about providing them
- recommends using a compilation database (as that seems to be the fashion)
- standardises formatting for `NOTE` (it was inconsistent before)
- states that the preferred installation method is `install.py` (rather than the full installation instructions)
- update the vim doc
- update the ycmd submodule
### ycmd update release note
- valloric/ycmd#678 - Bump Boost version to 1.63.0
- valloric/ycmd#686 - Update JediHTTP for Python 3.6 support
- valloric/ycmd#684 - Fix JavaScript identifier regex
- valloric/ycmd#680 - Automatically load a compilation database if found
[cont]: https://github.com/Valloric/YouCompleteMe/blob/master/CONTRIBUTING.md
[code]: https://github.com/Valloric/YouCompleteMe/blob/master/CODE_OF_CONDUCT.md
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2495)
<!-- Reviewable:end -->
[READY] Improve issue template - guide the user in completing it
# PR Prelude
Thank you for working on YCM! :)
**Please complete these steps and check these boxes (by putting an `x` inside
the brackets) _before_ filing your PR:**
- [X] I have read and understood YCM's [CONTRIBUTING][cont] document.
- [X] I have read and understood YCM's [CODE_OF_CONDUCT][code] document.
- [X] I have included tests for the changes in my PR. If not, I have included a
rationale for why I haven't.
> project administration-only change
- [X] **I understand my PR may be closed if it becomes obvious I didn't
actually perform all of these steps.**
# Why this change is necessary and useful
The issue template has worked very well and a larger number of reported issues include useful diagnostics. However, this is still far from ubiquitous.
I've noticed two recurring problems, that I've tried to solve with an update to the template:
- when reporting a build failure, not including the _entire_ `install.py` log or the command used to run it
- when reporting issues, stating that something doesn't work, but not stating what the expected behaviour was (this is a common human mistake).
Meanwhile, I've encouraged the reporter to fill in the sections by actually providing such sections in the template for them to fill in.
This is unlikely to get us to 100%, but hopefully another incremental increase in the quality of reports will come out of it.
[cont]: https://github.com/Valloric/YouCompleteMe/blob/master/CONTRIBUTING.md
[code]: https://github.com/Valloric/YouCompleteMe/blob/master/CODE_OF_CONDUCT.md
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2496)
<!-- Reviewable:end -->
[READY] Handle server exceptions from loading/ignoring extra conf
Since PR #2453, we don't catch anymore exceptions raised from loading or ignoring a `.ycm_extra_conf.py` file. For instance, loading this extra conf:
```python
raise Exception
```
will result in a Python traceback inside Vim. This is fixed by catching the `UnknownExtraConf` exception in its own block (as [it was before](48b7ccef76/python/ycm/client/event_notification.py (L64-L73))).
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2488)
<!-- Reviewable:end -->
[READY] Fix large file message appearing twice
In PR #2469, we moved the large file logic inside `s:OnBufferRead` but forgot this function was not only called on the `BufRead` event but also on the `FileType` one. This makes the large file message appears twice when opening such file. We fix that by not displaying the message when the `b:ycm_largefile` variable is already set.
Also, the message is now displayed with `PostVimMessage` to avoid the `Press ENTER or type command to continue` prompt.
Fixes#2485.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2487)
<!-- Reviewable:end -->
Vim automatically adds a space between strings when concatenating them
without the dot operator. This is not obvious so we always use the dot
operator.
Minor coding style changes when writing statements on multiple lines to
be more consistent.
[READY] Update ycmd
# PR Prelude
Thank you for working on YCM! :)
**Please complete these steps and check these boxes (by putting an `x` inside
the brackets) _before_ filing your PR:**
- [X] I have read and understood YCM's [CONTRIBUTING][cont] document.
- [X] I have read and understood YCM's [CODE_OF_CONDUCT][code] document.
- [X] I have included tests for the changes in my PR. If not, I have included a
rationale for why I haven't.
> ycmd tests itself
- [X] **I understand my PR may be closed if it becomes obvious I didn't
actually perform all of these steps.**
# Why this change is necessary and useful
A number of enhancements have been made in the server. They seem stable in my testing. Let's push them live now.
Includes all of the following PRs:
* Auto merge of valloric/ycmd#631 - vheon:refactor-request-wrap, r=micbou: [READY] Add 'first_filetype' to RequestWrap
* Auto merge of valloric/ycmd#634 - vheon:fix-previous-identifier, r=micbou: [READY] Avoid to wrap around when searching for previous identifier
* Auto merge of valloric/ycmd#635 - micbou:resource-warning, r=Valloric: [READY] Fix warnings when resetting process handle
* Auto merge of valloric/ycmd#637 - vheon:refactor-tags-file, r=micbou: [READY] Extract filtering of tags as a generator
* Auto merge of valloric/ycmd#639 - mixedCase:tern_0.20, r=micbou: Bump Tern to version 0.20
* Auto merge of valloric/ycmd#640 - vheon:test-identifier-completer, r=micbou: [READY] Additional test for the identifier completer
* Auto merge of valloric/ycmd#636 - puremourning:coverage-c++, r=micbou: [READY] Coverage for c++ code
* Auto merge of valloric/ycmd#641 - micbou:remove-tests-folder, r=vheon: [READY] Remove unused tests folderS
* Auto merge of valloric/ycmd#643 - micbou:javascript-identifiers, r=Valloric: [READY] Add regex for JavaScript and TypeScript identifiers
* Auto merge of valloric/ycmd#644 - micbou:flake8, r=Valloric: [READY] Fix flake8 errors
* Auto merge of valloric/ycmd#645 - micbou:global-config-exception, r=Valloric: [READY] Catch exceptions from global extra conf
* Auto merge of valloric/ycmd#649 - micbou:codecov, r=puremourning: [READY] Use codecov bash uploader on Travis
* Auto merge of valloric/ycmd#652 - micbou:remove-executable-mode, r=Valloric: [READY] Remove executable mode
* Auto merge of valloric/ycmd#650 - vheon:remove-unused-assert, r=micbou: Remove unused CustomAssert
* Auto merge of valloric/ycmd#651 - micbou:global-extra-conf-reuse, r=Valloric: [READY] Do not reload extra conf file
* Auto merge of valloric/ycmd#657 - micbou:gocode-submodule, r=Valloric: [READY] Update Gocode submodule (Fixes#2449)
* Auto merge of valloric/ycmd#665 - micbou:flake8, r=micbou: [READY] Do not disable Flake8 on Python 2.6 Travis but restrict its version
* Auto merge of valloric/ycmd#667 - haifengkao:SupportObjcImportModule, r=Valloric: Support objc import module
* Auto merge of valloric/ycmd#677 - puremourning:dev-flags, r=micbou: [READY] Enable dev flags when --enable-debug is supplied
[cont]: https://github.com/Valloric/YouCompleteMe/blob/master/CONTRIBUTING.md
[code]: https://github.com/Valloric/YouCompleteMe/blob/master/CODE_OF_CONDUCT.md
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2486)
<!-- Reviewable:end -->
[READY] Fix exception when response future is not set
An `AttributeError` exception is raised when `Done` is called before `Start` in the `CompletionRequest` and `EventNotification` classes because the `_response_future` attribute is not yet defined.
Fixes#2461.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2465)
<!-- Reviewable:end -->
[READY] Catch and log all server exceptions
A lot of different errors may occur when sending a request to ycmd and receiving its response:
- [any exception from the requests module](http://docs.python-requests.org/en/master/_modules/requests/exceptions/);
- exceptions from ycmd: `ServerError` and `UnknownExtraConf`;
- `RuntimeError` exception from invalid HMAC;
- and possibly others.
Thanks to PR #2430, we can now catch and log these exceptions.
I am marking this PR as WIP because I'd like to add tests but I am not sure on how to implement them. Should we use actual code for these tests or mock the response from the ycmd server?
Fixes#2216.
Fixes#2272.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/valloric/youcompleteme/2453)
<!-- Reviewable:end -->
This is another attempt at:
https://github.com/Valloric/YouCompleteMe/pull/2473
Which removed the apparently flawed attempt at "lazy" loading of YCM.
While it fails to get the load out of the critical startup path, it
*does* serve the useful purpose when starting up gvim of avoiding a
deadlock situation in gvim.
So, this time, we keep VimEnter, but only for the gui-starting case. We
update the comment to explain what is actually happening. And we can
keep the docs about how to defer loading.
