dart-lang / coverage Goto Github PK
View Code? Open in Web Editor NEWDart coverage data manipulation and formatting
Home Page: https://pub.dev/packages/coverage
License: BSD 3-Clause "New" or "Revised" License
coverage's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
Forkers
adam-singer cbracken srawlins johnryan-wf evanweible-wf ekweible bwu-dart-contributing jtmcdole jayudey-wf abarth dramos07 kbadova joeconwaystk zacharynewton-wk lrhn brammittendorff dalavancloud wouterhardeman dnfield simolus3 jonahwilliams bhanditz reborty a14n willdrach-wk ag-dart next-jorgehernandezramirez raieen michaelrfairhurst lig lukaslihotzki global-localhost global19 global19-atlassian-net iamriajul driky kyleferguson-wk star-lord-xiii maxim040592 jellynoone jensjoha tryweirdier lennoncastro natecoders derekxu16 coderdake classy-bear kturney fzyzcjycoverage's Issues
FormatException: Could not find an option named "packages"
So in order to format coverage, readme suggests to use command format_coverage --packages=<package_dir>/.packages -i <coverage json>
Running this command with coverage-0.7.8 instantly produces
Unhandled exception:
FormatException: Could not find an option named "packages".
#0 Parser.validate (package:args/src/parser.dart:247)
#1 Parser.parseLongOption (package:args/src/parser.dart:236)
#2 Parser.parse (package:args/src/parser.dart:77)
#3 ArgParser.parse (package:args/src/arg_parser.dart:133)
#4 parseArgs (http://localhost:51721/format_coverage.dart:115)
#5 main.<main_async_body> (http://localhost:51721/format_coverage.dart:28)
#6 Future.Future.microtask.<anonymous closure> (dart:async/future.dart:144)
#7 _microtaskLoop (dart:async/schedule_microtask.dart:41)
#8 _startMicrotaskLoop (dart:async/schedule_microtask.dart:50)
#9 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#10 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:149)
What is proper command to produce human-readable coverage report?
collect_coverage errors when isolates are spawned by the Dart script
I have a Dart library that does code generation that I am unable to reliably gather coverage from. The library has tests that rely on the output of an isolate spawned during the testing process.
The tests work as follows
- Run code generation step
- Spawn isolate which executes tests on the generated code which verifies they are working correctly.
- If isolate returns without error the code generation worked, if there's an error then the code generation failed.
It appears that when running dart --observe=8000 tests/all.dart that the isolate being spawned by the test is waiting for a connection to the observe port before exiting.
Running pub global run coverage:collect_coverage --port=8000 -o coverage.json then causes the test to complete but then the following error occurs
JsonRpcError: VM must be paused (101)
dart:async _Completer.completeError
package:coverage/src/devtools.dart 227 _Connection._handleResponse
===== asynchronous gap ===========================
dart:io _WebSocketImpl.listen
package:coverage/src/devtools.dart 184 _Connection._Connection
package:coverage/src/devtools.dart 191 _Connection.connect.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
package:coverage/src/devtools.dart _Connection.connect
package:coverage/src/devtools.dart 93 VMService.connectToVMWebsocket.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/devtools.dart VMService.connectToVMWebsocket
package:coverage/src/devtools.dart 89 VMService.connect.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/devtools.dart VMService.connect
package:coverage/src/collect.dart 17 collect.<async>.<fn>
package:coverage/src/util.dart 38 retry.<async>.<fn>.<async>
===== asynchronous gap ===========================
package:coverage retry.<async>.<fn>
package:coverage/src/util.dart 17 retry.<async>._withTimeout
package:coverage/src/util.dart 35 retry.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/util.dart retry
package:coverage/src/collect.dart 16 collect.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/collect.dart collect
http://localhost:50032/collect_coverage.dart 21 main.<async>.<fn>.<async>
===== asynchronous gap ===========================
package:stack_trace Chain.capture
http://localhost:50032/collect_coverage.dart 20 main.<async>
After running the command multiple times the coverage.json appears to be generated.
I'm not sure if this is a problem due to how dart is invoked or if coverage should be doing something different.
My setup is
sdk 1.14.0-dev.6.0
coverage 0.7.2
Windows 7
The repository I stumbled upon the issue is https://github.com/dogma-dart/dogma-codegen/.
A smaller use case is https://github.com/dogma-dart/dogma-codegen-test which is used by the above project to do this workflow. It has a set of tests as well that will not work with the coverage library.
I did have success with this environment for other projects so I think this just revolves around the isolate handling.
Example repo and coverage.json
I'm looking to add support for Dart to codecov.io (full disclosure: I'm the founder)
I'm not familiar with Dart and hoping someone can help show me an example coverage.json output file so that I can upload it to Codecov. If you have a very lean example repo I can show customers that would be great!
Thanks ~ Steve
Strong mode issues
> dartanalyzer --strong . | sort | uniq
4 errors, 14 warnings and 2 hints found.
Analyzing [.]...
[error] Field declaration Resolver.failed cannot be overridden in BazelResolver. (/usr/local/google/home/hcameron/github/coverage/lib/src/resolver.dart, line 93, col 3)
[error] The argument type 'List' cannot be assigned to the parameter type 'Iterable<File>'. (/usr/local/google/home/hcameron/github/coverage/bin/format_coverage.dart, line 44, col 36)
[hint] Unused import (/usr/local/google/home/hcameron/github/coverage/lib/src/formatter.dart, line 4, col 8)
[warning] Unsound implicit cast from dynamic to List<Map<dynamic, dynamic>> (/usr/local/google/home/hcameron/github/coverage/lib/src/hitmap.dart, line 74, col 31)
[warning] Unsound implicit cast from dynamic to List<Map<dynamic, dynamic>> (/usr/local/google/home/hcameron/github/coverage/test/lcov_test.dart, line 168, col 29)
[warning] Unsound implicit cast from dynamic to List<String> (/usr/local/google/home/hcameron/github/coverage/bin/format_coverage.dart, line 186, col 18)
[warning] Unsound implicit cast from Future<dynamic> to Future<Null> (/usr/local/google/home/hcameron/github/coverage/test/test_util.dart, line 34, col 10)
[warning] Unsound implicit cast from List<dynamic> to List<Map<dynamic, dynamic>> (/usr/local/google/home/hcameron/github/coverage/test/collect_coverage_test.dart, line 62, col 31)
[warning] Unsound implicit cast from List<dynamic> to List<Map<dynamic, dynamic>> (/usr/local/google/home/hcameron/github/coverage/test/run_and_collect_test.dart, line 47, col 31)
Cannot resolve symbolic links,
Hi I have a few problems with formating my coverage.json file... Any ideas or points would be most helpful.
Rays-iMac-2:goongo rayk$ format_coverage -s /usr/local/opt/dart/libexec -p /Users/rayk/code/goongo -i coverage.json
Unhandled exception:
Uncaught Error: FileSystemException: Cannot resolve symbolic links, path = '/usr/local/opt/dart/libexec/lib/_internal/list.dart' (OS Error: No such file or directory, errno = 2)
Stack Trace:
#0 FileSystemEntity._throwIfError (dart:io/file_system_entity.dart:671)
#1 FileSystemEntity.resolveSymbolicLinksSync (dart:io/file_system_entity.dart:338)
#2 Resolver.resolveSymbolicLinks (package:coverage/src/resolver.dart:66)
#3 Resolver.resolve (package:coverage/src/resolver.dart:44)
#4 PrettyPrintFormatter.format.<anonymous closure> (package:coverage/src/formatter.dart:55)
#5 Future.forEach.<anonymous closure>.<anonymous closure> (dart:async/future.dart:336)
#6 Future.Future.sync (dart:async/future.dart:168)
#7 Future.forEach.<anonymous closure> (dart:async/future.dart:336)
#8 Future.Future.sync (dart:async/future.dart:168)
#9 Future.doWhile.<anonymous closure> (dart:async/future.dart:361)
#10 _RootZone.runUnaryGuarded (dart:async/zone.dart:1093)
#11 _RootZone.bindUnaryCallback.<anonymous closure> (dart:async/zone.dart:1122)
#12 _RootZone.runUnary (dart:async/zone.dart:1155)
#13 _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:494)
#14 _Future._propagateToListeners (dart:async/future_impl.dart:577)
#15 _Future._completeWithValue (dart:async/future_impl.dart:368)
#16 _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:422)
#17 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#18 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#19 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#20 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:143)
#0 _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:886)
#1 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#2 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#3 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:143)
Rays-iMac-2:goongo rayk$ ***And when I don't pass the symlink for the Sdk I get:
Rays-iMac-2:goongo rayk$ format_coverage -p /Users/rayk/code/goongo -i coverage.json
Unhandled exception:
Uncaught Error: FileSystemException: Cannot resolve symbolic links, path = '/Volumes/data/b/build/slave/dartium-mac-full-stable/build/src/dart/sdk/lib/html/html_common/css_class_set.dart' (OS Error: No such file or directory, errno = 2)
Stack Trace:
#0 FileSystemEntity._throwIfError (dart:io/file_system_entity.dart:671)
#1 FileSystemEntity.resolveSymbolicLinksSync (dart:io/file_system_entity.dart:338)
#2 Resolver.resolveSymbolicLinks (package:coverage/src/resolver.dart:66)
#3 Resolver.resolve (package:coverage/src/resolver.dart:55)
#4 PrettyPrintFormatter.format.<anonymous closure> (package:coverage/src/formatter.dart:55)
#5 Future.forEach.<anonymous closure>.<anonymous closure> (dart:async/future.dart:336)
#6 Future.Future.sync (dart:async/future.dart:168)
#7 Future.forEach.<anonymous closure> (dart:async/future.dart:336)
#8 Future.Future.sync (dart:async/future.dart:168)
#9 Future.doWhile.<anonymous closure> (dart:async/future.dart:361)
#10 _RootZone.runUnaryGuarded (dart:async/zone.dart:1093)
#11 _RootZone.bindUnaryCallback.<anonymous closure> (dart:async/zone.dart:1122)
#12 _RootZone.runUnary (dart:async/zone.dart:1155)
#13 _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:494)
#14 _Future._propagateToListeners (dart:async/future_impl.dart:577)
#15 _Future._completeWithValue (dart:async/future_impl.dart:368)
#16 _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:422)
#17 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#18 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#19 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#20 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:143)
#0 _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:886)
#1 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#2 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#3 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:143)
collect_coverage does nothing
I'm running tests in content shell and trying to get a coverage report but can't seem to get coverage tool to do anything at all.
I start the test runner like so:
$ pub run test --pause-after-load test/editor/editor_test.dart
Warning: Debugging is currently unsupported on the Dart VM.
00:00 +0: [Dartium Content Shell] loading test/editor/model_test.dart
Observatory URL: http://127.0.0.1:42350/
Remote debugger URL: http://localhost:45598/devtools/inspector.html?ws=localhost:45598/devtools/page/3319C366-C0AB-490D-AB52-F494309775E8
The test runner is paused. Open the remote debugger or the Observatory and set breakpoints. Once
you're finished, return to this terminal and press Enter.
Once I get the debugger URL I start the coverage tool:
collect_coverage --uri="http://localhost:45598/devtools/inspector.html?ws=localhost:45598/devtools/page/3319C366-C0AB-490D-AB52-F494309775E8"
I then go back to the test runner and press Enter.
The tests all pass and the test runner exits but nothing is ever printed in the collect_coverage tool. There is no errors or any other messages. I also don't see any flags to make collect_coverage a little more verbose, maybe there would be clues about the problem.
Am I doing something wrong with how I use coverage or is this a bug?
Dart VM: 1.22.0-dev.10.1 (Sat Jan 28 19:46:18 2017) on "linux_x64"
Content Shell: linux-x64-dev-1.22.0-dev.10.1.0
bin/ scripts should include "dart" in the name to aid readability of scripts that use them
Coverage report miss for external functions
I'm seeing an issue where dart coverage is reporting lines as not being hit when they clearly are being hit. It involves dart bindings to external javascript functions, so I suspect that has something to do with it.
This is an example of what my class I'm testing looks like (bound to external javascript code)
@JS()
@anonymous
class TestMe {
external factory TestMe();
external int returnFive();
}
And my test case. I verified that this test is being run and passing.
test('Test ReturnFive returns 5', () {
var testClass = new TestMe();
expect(testClass.returnFive(), equals(5));
});
When I run coverage for that test though, it reports the line for returnFive as not being hit
dart: 1.21.1
coverage: 0.7.3 (also tested with 0.9.1 and issue still occurs)
Single command to collect and format coverage
Add --json, --lcov, --pretty-print (default) flags to specify output. Requires adding --sdk-root and --package-root flags.
Blocked by #7.
Remove 1.10 fallback logic once Dart 1.11 goes stable
Use service library from Observatory
Service library in Observatory can handle all communication over the service (to content_shell, standalone, and Chrome VM) and provides model objects for responses. It should allow you to drop a bunch of code.
coverage_format fails with an exception dart-sdk/lib/_internal/list.dart
(master) $ format_coverage --package-root=. -i coverage.json
Unhandled exception:
Uncaught Error: FileSystemException: Cannot resolve symbolic links, path = '/home/myuser/darteditor/dart-sdk/lib/_internal/list.dart' (OS Error: No such file or directory, errno = 2)
Stack Trace:
#0 FileSystemEntity._throwIfError (dart:io/file_system_entity.dart:671)
#1 FileSystemEntity.resolveSymbolicLinksSync (dart:io/file_system_entity.dart:338)
#2 Resolver.resolveSymbolicLinks (package:coverage/src/resolver.dart:66:62)
#3 Resolver.resolve (package:coverage/src/resolver.dart:44:34)
#4 PrettyPrintFormatter.format.<anonymous closure> (package:coverage/src/formatter.dart:55:33)
#5 Future.forEach.<anonymous closure>.<anonymous closure> (dart:async/future.dart:336)
#6 Future.Future.sync (dart:async/future.dart:168)
#7 Future.forEach.<anonymous closure> (dart:async/future.dart:336)
#8 Future.Future.sync (dart:async/future.dart:168)
#9 Future.doWhile.<anonymous closure> (dart:async/future.dart:361)
#10 _RootZone.runUnaryGuarded (dart:async/zone.dart:1093)
#11 _RootZone.bindUnaryCallback.<anonymous closure> (dart:async/zone.dart:1122)
#12 _RootZone.runUnary (dart:async/zone.dart:1155)
#13 _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:484)
#14 _Future._propagateToListeners (dart:async/future_impl.dart:567)
#15 _Future._completeWithValue (dart:async/future_impl.dart:358)
#16 _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:412)
#17 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#18 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#19 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#20 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:143)
#0 _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:886)
#1 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#2 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#3 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:143)
The file can be found in
'/home/myuser/darteditor/dart-sdk/lib/internal/list.dart'
instead of
'/home/myuser/darteditor/dart-sdk/lib/_internal/list.dart'
Where does the underline in _internal come from?
Request: add support for checked mode in `runAndCollect` function
I usually run my tests in checked mode in order to catch every possible error in continuous integration.
My tests are based on the test package, and their runner has the checked mode enabled by default (see files vm/platform.dart, runner/browser/content_shell.dart and runner/browser/dartium.dart).
I'm developing a Grinder plugin that collects the code coverage using the runAndCollect function, and I would like that my tests to have the same behavior with or without coverage.
So it would be cool if an optional bool checked parameter were added.
Support the coverage protocol in 1.11
Unable to get collect_coverage to work with Dartium; always quits too soon
A some point, this script worked, but intermittently. Since upgrading to Windows 8.1, it just doesn't work at all. I've tried everything, including --wait-paused, kickin off Dartium both before and after collect_coverage, but collect_coverage always quits as soon as Dartium loads, and doesn't wait for the tests to finish executing.
What is the correct way to make this work?
Note: I'm usig 0.5.0, because 0.6.0 just crashes immediately (I'll raise another case for that shortly).
$testInstance = "http://dart-tests/"
$dartiumPath = "C:\Work\Utilities\dartium\chrome.exe"
$dartiumProfilePath = "C:\Work\Utilities\dartium-profile"
$timeoutSeconds = 10
$debuggerPort = 8091
$dartSDK = "%DART_SDK%"
# Start listening for coverage
Write-Host "Starting collect_coverage..."
$proc = Start-Process pub -ArgumentList ("global", "run", "coverage:collect_coverage", "--port=$debuggerPort", "-o coverage/coverage.json", "--connect-timeout=$timeoutSeconds") -PassThru
# Start Dartium and run the tests
Write-Host "Starting Dartium to run tests..."
$dartium = Start-Process $dartiumPath -ArgumentList ("--remote-debugging-port=$debuggerPort", "--user-data-dir=$dartiumProfilePath", "--disable-web-security", $testInstance) -PassThru
# Wait long enough for the tests to finish
Write-Host "Waiting for tests to finish..."
sleep ($timeoutSeconds + 1)
# Kill dartium
Write-Host "Killing Dartiumn processes..."
$dartium | kill
# Wait for coverage to end
Write-Host "Waiting for collect_coverage to finish..."
$proc.WaitForExit()
# Format coverage
Write-Host "Formatting coverage..."
dart tool\format_coverage.dartLarge memory consumption when collecting coverage for many isolates
We have tests in Aqueduct that spawn many isolates (>30). The current coverage strategy pauses isolates at exit and waits for tests/collection to finish before resuming each isolate so that it completes. Memory consumption gets large - I've seen it reach 10gb. We've split up tests that spawn a lot of isolates into separate files, but boxes that run automated test/coverage are still failing with out of memory exceptions.
As a trial, I forked this repository and added code to resume each isolate after it was collected. This did reduce memory consumption down to a negligible amount, but created failures later in the process. With that modification, the coverage script will simply exit after a period of time with no exception, but without the work being completed. I'm afraid I've reached the limit of my understanding here.
Simplify coverage collection
Rather than forcing the user to start a VM/Dartium instance then connect to it separately via collect_coverage.dart, support launching a Dart VM/Dartium content-shell, collecting, and emitting coverage in a single step:
e.g. dart bin/collect_coverage foo_test.dart
Add tests to exercise collecting coverage from content-shell
Currently the test suite for this project only exercises collecting coverage from a dart application running on the VM. This does not cover collecting coverage from content-shell.
I bring this up because Travis CI always uses the latest version of Dart, but at times there's an issue with the version of content-shell that it uses for CI builds (see travis-ci/travis-ci#4406). If this package had tests to exercise content-shell coverage collection, those issues would potentially be caught by the CI build.
collect() hangs if the process it's collecting from dies mid-collection
Collecting coverage from dartium or content_shell fails due to JsonRpcError
This seems to be a problem ever since Dart SDK 1.11.
Usage:
$ dartium --remote-debugging-port=8024 test/example_test.html
[87984:1291:0806/145141:ERROR:renderer_main.cc(207)] Running without renderer sandbox
[87985:1291:0806/145141:ERROR:renderer_main.cc(207)] Running without renderer sandbox
Observatory listening on http://127.0.0.1:54563
Observatory listening on http://127.0.0.1:54564
$ pub run coverage:collect_coverage --port=54563 -o coverage.json --resume-isolates
JsonRpcError: VM must be paused (101)
Side Note: Is
--remote-debugging-portnecessary? As I understand it, observatory picks its own ports for each tab and there's no way to force it to use a certain port (at least not yet).
Full Error Stacktrace:
JsonRpcError: VM must be paused (101)
dart:async _Completer.completeError
package:coverage/src/devtools.dart 227 _Connection._handleResponse
===== asynchronous gap ===========================
dart:io _WebSocketImpl.listen
package:coverage/src/devtools.dart 184 _Connection._Connection
package:coverage/src/devtools.dart 191 _Connection.connect.<async>
===== asynchronous gap ===========================
dart:async _Future.then
package:coverage/src/devtools.dart 190 _Connection.connect.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/devtools.dart _Connection.connect
package:coverage/src/devtools.dart 93 VMService.connectToVMWebsocket.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/devtools.dart VMService.connectToVMWebsocket
package:coverage/src/devtools.dart 89 VMService.connect.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/devtools.dart VMService.connect
package:coverage/src/collect.dart 17 collect.<async>.<fn>
package:coverage/src/util.dart 38 retry.<async>.<fn>.<async>
===== asynchronous gap ===========================
package:coverage retry.<async>.<fn>
package:coverage/src/util.dart 17 retry.<async>._withTimeout
package:coverage/src/util.dart 35 retry.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/util.dart retry
package:coverage/src/collect.dart 16 collect.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/collect.dart collect
http://localhost:51502/collect_coverage.dart 21 main.<async>.<fn>.<async>
===== asynchronous gap ===========================
package:stack_trace Chain.capture
http://localhost:51502/collect_coverage.dart 20 main.<async>
Support loading script source from Observatory
For use with --pretty-print formatting.
Small hitch: generally people are more interested on coverage on their checked in sources. In the case where pub (barback) transformers are used, the VM is collecting on (and would be pretty-printing) the transformed code. Transformers don't emit source maps, but by convention, generally tack new lines of code onto the end of existing lines, and replace deleted lines with a blank line so as to keep the line mappings identical.
Default to checked mode in runAndCollect()
Dart 1.14.1 - collecting coverage from dartium and content_shell fails with JsonRpcError: Extension error (105)
Since Dart 1.14.1 was released, collecting coverage from the browser (both content_shell and dartium) seems to be broken.
Package setup for reduced test case: https://gist.github.com/evanweible-wf/fed84c33e194fccba246
vm (working)
$ dart --observe=8444 test/vm_test.dart
$ pub run coverage:collect_coverage --port=8444 -o coverage.json --resume-isolates
content_shell
$ pub serve test
$ content_shell http://localhost:8080/browser_test.html
[36008:1291:0205/132456:544536108451496:ERROR:renderer_main.cc(207)] Running without renderer sandbox
Observatory listening on http://127.0.0.1:50160
[36006:2575:0205/132457:544536824440195:INFO:CONSOLE(0)] "Observatory listening at http://127.0.0.1:50160/", source: http://localhost:8080/browser_test.html (0)
[36006:2575:0205/132457:544537018471014:INFO:CONSOLE(1)] "00:00 +0: test browserA()", source: (1)
[36006:2575:0205/132457:544537037686188:INFO:CONSOLE(1)] "browserA", source: (1)
[36006:2575:0205/132457:544537045773233:INFO:CONSOLE(1)] "00:00 +1: All tests passed!", source: (1)
$ pub run coverage:collect_coverage --port=50160 -o coverage.json
JsonRpcError: Extension error (105)
dart:async _Completer.completeError
package:coverage/src/devtools.dart 227 _Connection._handleResponse
===== asynchronous gap ===========================
dart:io _WebSocketImpl.listen
package:coverage/src/devtools.dart 184 _Connection._Connection
package:coverage/src/devtools.dart 191 _Connection.connect.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
package:coverage/src/devtools.dart _Connection.connect
package:coverage/src/devtools.dart 93 VMService.connectToVMWebsocket.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/devtools.dart VMService.connectToVMWebsocket
package:coverage/src/devtools.dart 89 VMService.connect.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/devtools.dart VMService.connect
package:coverage/src/collect.dart 17 collect.<async>.<fn>
package:coverage/src/util.dart 38 retry.<async>.<fn>.<async>
===== asynchronous gap ===========================
package:coverage retry.<async>.<fn>
package:coverage/src/util.dart 17 retry.<async>._withTimeout
package:coverage/src/util.dart 35 retry.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/util.dart retry
package:coverage/src/collect.dart 16 collect.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/collect.dart collect
http://localhost:64284/collect_coverage.dart 21 main.<async>.<fn>.<async>
===== asynchronous gap ===========================
package:stack_trace Chain.capture
http://localhost:64284/collect_coverage.dart 20 main.<async>
Also tried with the
--resume-isolatesflag, same result.
dartium
$ pub serve test
$ dartium http://localhost:8080/browser_test.html
[36889:1291:0205/132723:ERROR:renderer_main.cc(207)] Running without renderer sandbox
Observatory listening on http://127.0.0.1:50187
[36890:1291:0205/132724:ERROR:renderer_main.cc(207)] Running without renderer sandbox
Observatory listening on http://127.0.0.1:50188
[36891:1291:0205/132725:ERROR:renderer_main.cc(207)] Running without renderer sandbox
Observatory listening on http://127.0.0.1:50191
Note the 3 different observatory ports.
1st observatory port:
$ pub run coverage:collect_coverage --port=50187 -o coverage.json
JsonRpcError: Extension error (105)
dart:async _Completer.completeError
package:coverage/src/devtools.dart 227 _Connection._handleResponse
===== asynchronous gap ===========================
dart:io _WebSocketImpl.listen
package:coverage/src/devtools.dart 184 _Connection._Connection
package:coverage/src/devtools.dart 191 _Connection.connect.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
package:coverage/src/devtools.dart _Connection.connect
package:coverage/src/devtools.dart 93 VMService.connectToVMWebsocket.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/devtools.dart VMService.connectToVMWebsocket
package:coverage/src/devtools.dart 89 VMService.connect.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/devtools.dart VMService.connect
package:coverage/src/collect.dart 17 collect.<async>.<fn>
package:coverage/src/util.dart 38 retry.<async>.<fn>.<async>
===== asynchronous gap ===========================
package:coverage retry.<async>.<fn>
package:coverage/src/util.dart 17 retry.<async>._withTimeout
package:coverage/src/util.dart 35 retry.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/util.dart retry
package:coverage/src/collect.dart 16 collect.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/collect.dart collect
http://localhost:64284/collect_coverage.dart 21 main.<async>.<fn>.<async>
===== asynchronous gap ===========================
package:stack_trace Chain.capture
http://localhost:64284/collect_coverage.dart 20 main.<async>
2nd observatory port:
$ pub run coverage:collect_coverage --port=50188 -o coverage.json
Empty coverage.
3rd observatory port:
$ pub run coverage:collect_coverage --port=50191 -o coverage.json
[hangs]
Also tried these with the
--resume-isolatesflag, same results.
Remove hacked `vm_service_client` once package support coverage report
Hack was added here f3dfba4
Remove once dart-archive/vm_service_client#6 lands in pkg/vm_service_client to add getSourceReport
Support new packages file in dart 1.12
The format coverage task takes a --package-root option to discover packages. This fails in projects using dart 1.12 with the --no-package-symlinks flag.
Add Branch and Function coverage to lcov output.
It would be nice to have coverage data on branches and function/methods.
afaict the coverage.json just has a list of lines hit, but no data on branches and functions. Also, there is no code in LcovFormatter that generates branch and function lines.
Side note, I'm not totally sure of the coverage.json format, but it seems like there are a lot of extraneous zeros in the lines hit:
{
"type": "CodeCoverage",
"coverage": [{
"source": "dart:core-patch/type_patch.dart",
"script": {
"type": "@Script",
"fixedId": true,
"id": "libraries/1/scripts/dart%3Acore-patch%2Ftype_patch.dart",
"uri": "dart:core-patch/type_patch.dart",
"_kind": "library"
},
"hits": []
}, {
"source": "dart:core/errors.dart",
"script": {
"type": "@Script",
"fixedId": true,
"id": "libraries/1/scripts/dart%3Acore%2Ferrors.dart",
"uri": "dart:core/errors.dart",
"_kind": "library"
},
"hits": [517, 0, 518, 0, 208, 0, 219, 0, 240, 0, 267, 2, 268, 0, 285, 0, 287, 0, 289, 0, 314, 2, 319, 2, 316, 0, 321, 0, 334, 1, 341, 0, 342, 0, 343, 0, 346, 0, 347, 0, 348, 0, 349, 0, 350, 0, 354, 0, 385, 0, 386, 0, 391, 0, 396, 0, 399, 0, 402, 0, 533, 0, 536, 0, 540, 0, 424, 0, 485, 0, 486, 0, 554, 0, 145, 0, 164, 0, 172, 0, 175, 0, 180, 0, 181, 0, 183, 0, 184, 0, 185, 0, 187, 0, 188, 0, 189, 0, 502, 0, 503, 0, 504, 0, 505, 0, 69, 0, 78, 0, 79, 0, 81, 0, 82, 0, 84, 0, 416, 0, 99, 0, 569, 0, 570, 0, 572, 0, 546, 0]
}, {
"source": "dart:core-patch/immutable_map.dart",
"script": {
"type": "@Script",
"fixedId": true,
"id": "libraries/1/scripts/dart%3Acore-patch%2Fimmutable_map.dart",
"uri": "dart:core-patch/immutable_map.dart",
"_kind": "library"
},
"hits": [137, 0, 140, 0, 141, 0, 142, 0, 143, 0, 146, 0, 147, 0, 151, 0, 90, 0, 93, 0, 96, 0, 101, 0, 104, 0, 107, 0, 115, 0, 118, 0, 119, 0, 120, 0, 121, 0, 124, 0, 125, 0, 129, 0, 10, 1, 17, 5, 18, 3, 19, 3, 26, 3, 29, 1, 32, 0, 36, 4, 37, 6, 42, 0, 46, 0, 50, 0, 51, 0, 59, 0, 60, 0, 68, 0, 72, 0, 76, 0, 80, 0, 84, 0]
},
...Add examples directory/targets
Dart SDK 1.20.x breaks coverage
With the latest SDK release, this package's tests are failing and existing consumptions of this package are broken.
$ pub run test
00:08 +17 -1: test/lcov_test.dart: LcovFormatter format()
Expected: contains '/Users/evanweible/dev/workiva/cp/coverage/lib/src/util.dart'
Actual: 'SF:/Users/evanweible/dev/workiva/cp/coverage/test/test_files/test_app.dart\n'
'DA:13,2\n'
'DA:14,2\n'
'DA:15,1\n'
'DA:16,2\n'
'DA:22,1\n'
'DA:25,4\n'
'DA:26,2\n'
'DA:27,2\n'
'DA:29,2\n'
'DA:31,1\n'
'DA:35,3\n'
'DA:36,1\n'
'DA:37,1\n'
'DA:40,1\n'
'DA:44,0\n'
'end_of_record\n'
'SF:/Users/evanweible/dev/workiva/cp/coverage/test/test_files/test_app_isolate.dart\n'
'DA:12,1\n'
'DA:14,1\n'
'DA:16,3\n'
'DA:18,1\n'
'end_of_record\n'
''
package:test expect
test/lcov_test.dart 53:7 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 59:5 LcovFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _Completer.completeError
test/lcov_test.dart 54:6 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 59:5 LcovFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
test/lcov_test.dart main.<fn>.<fn>
00:11 +17 -2: test/lcov_test.dart: LcovFormatter format() includes files in reportOn list
Expected: contains '/Users/evanweible/dev/workiva/cp/coverage/lib/src/util.dart'
Actual: 'SF:/Users/evanweible/dev/workiva/cp/coverage/test/test_files/test_app.dart\n'
'DA:13,2\n'
'DA:14,2\n'
'DA:15,1\n'
'DA:16,2\n'
'DA:22,1\n'
'DA:25,4\n'
'DA:26,2\n'
'DA:27,2\n'
'DA:29,2\n'
'DA:31,1\n'
'DA:35,3\n'
'DA:36,1\n'
'DA:37,1\n'
'DA:40,1\n'
'DA:44,0\n'
'end_of_record\n'
'SF:/Users/evanweible/dev/workiva/cp/coverage/test/test_files/test_app_isolate.dart\n'
'DA:12,1\n'
'DA:14,1\n'
'DA:16,3\n'
'DA:18,1\n'
'end_of_record\n'
''
package:test expect
test/lcov_test.dart 66:7 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 59:5 LcovFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _Completer.completeError
test/lcov_test.dart 67:6 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 59:5 LcovFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
test/lcov_test.dart main.<fn>.<fn>
00:14 +17 -3: test/lcov_test.dart: LcovFormatter format() excludes files not in reportOn list
Expected: contains '/Users/evanweible/dev/workiva/cp/coverage/lib/src/util.dart'
Actual: ''
package:test expect
test/lcov_test.dart 79:7 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 59:5 LcovFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _Completer.completeError
test/lcov_test.dart 80:6 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 59:5 LcovFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
test/lcov_test.dart main.<fn>.<fn>
00:17 +17 -4: test/lcov_test.dart: LcovFormatter format() uses paths relative to basePath
Expected: contains 'src/util.dart'
Actual: 'SF:../test/test_files/test_app.dart\n'
'DA:13,2\n'
'DA:14,2\n'
'DA:15,1\n'
'DA:16,2\n'
'DA:22,1\n'
'DA:25,4\n'
'DA:26,2\n'
'DA:27,2\n'
'DA:29,2\n'
'DA:31,1\n'
'DA:35,3\n'
'DA:36,1\n'
'DA:37,1\n'
'DA:40,1\n'
'DA:44,0\n'
'end_of_record\n'
'SF:../test/test_files/test_app_isolate.dart\n'
'DA:12,1\n'
'DA:14,1\n'
'DA:16,3\n'
'DA:18,1\n'
'end_of_record\n'
''
package:test expect
test/lcov_test.dart 92:7 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 59:5 LcovFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _Completer.completeError
test/lcov_test.dart 93:6 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 59:5 LcovFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
test/lcov_test.dart main.<fn>.<fn>
00:20 +17 -5: test/lcov_test.dart: PrettyPrintFormatter format()
Expected: contains '/Users/evanweible/dev/workiva/cp/coverage/lib/src/util.dart'
Actual: '/Users/evanweible/dev/workiva/cp/coverage/test/test_files/test_app.dart\n'
' |// Copyright (c) 2015, the Dart project authors. Please see the AUTHORS file\n'
' |// for details. All rights reserved. Use of this source code is governed by a\n'
' |// BSD-style license that can be found in the LICENSE file.\n'
' |\n'
' |import \'dart:isolate\';\n'
' |\n'
' |// explicitly using a package import to validate hitmap coverage of packages\n'
' |import \'package:coverage/src/util.dart\';\n'
' |\n'
' |import \'test_app_isolate.dart\';\n'
' |\n'
' |main() async {\n'
' 2| for (var i = 0; i < 10; i++) {\n'
' 2| for (var j = 0; j < 10; j++) {\n'
' 1| var sum = usedMethod(i, j);\n'
' 2| if (sum != (i + j)) {\n'
' | throw \'bad method!\';\n'
' | }\n'
' | }\n'
' | }\n'
' |\n'
' 1| ReceivePort port = new ReceivePort();\n'
' |\n'
' | Isolate isolate =\n'
' 4| await Isolate.spawn(isolateTask, [port.sendPort, 1, 2], paused: true);\n'
' 2| isolate.addOnExitListener(port.sendPort);\n'
' 2| isolate.resume(isolate.pauseCapability);\n'
' |\n'
' 2| var value = await port.first;\n'
' |\n'
' 1| if (value != 3) {\n'
' | throw \'expected 3!\';\n'
' | }\n'
' |\n'
' 3| var result = await retry(() async => 42, const Duration(seconds: 1));\n'
' 1| print(result);\n'
' 1|}\n'
' |\n'
' |int usedMethod(int a, int b) {\n'
' 1| return a + b;\n'
' |}\n'
' |\n'
' |int unusedMethod(int a, int b) {\n'
' 0| return a - b;\n'
' |}\n'
'/Users/evanweible/dev/workiva/cp/coverage/test/test_files/test_app_isolate.dart\n'
' |// Copyright (c) 2015, the Dart project authors. Please see the AUTHORS file\n'
' |// for details. All rights reserved. Use of this source code is governed by a\n'
' |// BSD-style license that can be found in the LICENSE file.\n'
' |\n'
' |import \'dart:io\';\n'
' |import \'dart:isolate\';\n'
' |\n'
' |/// The number of covered lines is tested and expected to be 4.\n'
' |///\n'
' |/// If you modify this method, you may have to update the tests!\n'
' |void isolateTask(List threeThings) {\n'
' 1| sleep(const Duration(milliseconds: 500));\n'
' |\n'
' 1| SendPort port = threeThings.first;\n'
' |\n'
' 3| var sum = threeThings[1] + threeThings[2];\n'
' |\n'
' 1| port.send(sum);\n'
' |}\n'
''
package:test expect
test/lcov_test.dart 107:7 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 108:5 PrettyPrintFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _Completer.completeError
test/lcov_test.dart 120:6 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 108:5 PrettyPrintFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
test/lcov_test.dart main.<fn>.<fn>
00:23 +17 -6: test/lcov_test.dart: PrettyPrintFormatter format() includes files in reportOn list
Expected: contains '/Users/evanweible/dev/workiva/cp/coverage/lib/src/util.dart'
Actual: '/Users/evanweible/dev/workiva/cp/coverage/test/test_files/test_app.dart\n'
' |// Copyright (c) 2015, the Dart project authors. Please see the AUTHORS file\n'
' |// for details. All rights reserved. Use of this source code is governed by a\n'
' |// BSD-style license that can be found in the LICENSE file.\n'
' |\n'
' |import \'dart:isolate\';\n'
' |\n'
' |// explicitly using a package import to validate hitmap coverage of packages\n'
' |import \'package:coverage/src/util.dart\';\n'
' |\n'
' |import \'test_app_isolate.dart\';\n'
' |\n'
' |main() async {\n'
' 2| for (var i = 0; i < 10; i++) {\n'
' 2| for (var j = 0; j < 10; j++) {\n'
' 1| var sum = usedMethod(i, j);\n'
' 2| if (sum != (i + j)) {\n'
' | throw \'bad method!\';\n'
' | }\n'
' | }\n'
' | }\n'
' |\n'
' 1| ReceivePort port = new ReceivePort();\n'
' |\n'
' | Isolate isolate =\n'
' 4| await Isolate.spawn(isolateTask, [port.sendPort, 1, 2], paused: true);\n'
' 2| isolate.addOnExitListener(port.sendPort);\n'
' 2| isolate.resume(isolate.pauseCapability);\n'
' |\n'
' 2| var value = await port.first;\n'
' |\n'
' 1| if (value != 3) {\n'
' | throw \'expected 3!\';\n'
' | }\n'
' |\n'
' 3| var result = await retry(() async => 42, const Duration(seconds: 1));\n'
' 1| print(result);\n'
' 1|}\n'
' |\n'
' |int usedMethod(int a, int b) {\n'
' 1| return a + b;\n'
' |}\n'
' |\n'
' |int unusedMethod(int a, int b) {\n'
' 0| return a - b;\n'
' |}\n'
'/Users/evanweible/dev/workiva/cp/coverage/test/test_files/test_app_isolate.dart\n'
' |// Copyright (c) 2015, the Dart project authors. Please see the AUTHORS file\n'
' |// for details. All rights reserved. Use of this source code is governed by a\n'
' |// BSD-style license that can be found in the LICENSE file.\n'
' |\n'
' |import \'dart:io\';\n'
' |import \'dart:isolate\';\n'
' |\n'
' |/// The number of covered lines is tested and expected to be 4.\n'
' |///\n'
' |/// If you modify this method, you may have to update the tests!\n'
' |void isolateTask(List threeThings) {\n'
' 1| sleep(const Duration(milliseconds: 500));\n'
' |\n'
' 1| SendPort port = threeThings.first;\n'
' |\n'
' 3| var sum = threeThings[1] + threeThings[2];\n'
' |\n'
' 1| port.send(sum);\n'
' |}\n'
''
package:test expect
test/lcov_test.dart 133:7 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 108:5 PrettyPrintFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _Completer.completeError
test/lcov_test.dart 134:6 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 108:5 PrettyPrintFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
test/lcov_test.dart main.<fn>.<fn>
00:26 +17 -7: test/lcov_test.dart: PrettyPrintFormatter format() excludes files not in reportOn list
Expected: contains '/Users/evanweible/dev/workiva/cp/coverage/lib/src/util.dart'
Actual: ''
package:test expect
test/lcov_test.dart 147:7 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 108:5 PrettyPrintFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _Completer.completeError
test/lcov_test.dart 148:6 main.<fn>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/formatter.dart 108:5 PrettyPrintFormatter.format.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
test/lcov_test.dart main.<fn>.<fn>
00:26 +17 -7: Some tests failed.
Missing counts?
252 : : if (ifAbsent != null) {
253 : 0 : if (!current.isValue) {
254 : 0 : _modCount++;
255 : 2 : current.value = ifAbsent();
256 : : }
257 : : } else {
255 is counted twice, but the conditional and modcount lines are not. Maybe its a VM issue?
Needs to be updated to support Dart 1.16 service protocol
I haven't upgraded to 1.16 yet, but Travis builds suddenly started to fail with following error:
$ pub global run coverage:collect_coverage --port=8111 -o coverage.json --resume-isolates
JSON-RPC error -32601 (method not found): Method not found
package:json_rpc_2/src/client.dart 99 Client.sendRequest
package:json_rpc_2/src/peer.dart 94 Peer.sendRequest
package:coverage/src/vm_service_client.dart 198 VMServiceClient.getCoverage.<async>
===== asynchronous gap ===========================
dart:async Future.Future.microtask
package:coverage/src/vm_service_client.dart VMServiceClient.getCoverage
package:coverage/src/collect.dart 41 _getAllCoverage.<async>
dart:async _SyncCompleter.complete
package:vm_service_client/src/isolate.dart 232 VMIsolateRef.load.<async>
dart:async _SyncCompleter.complete
package:vm_service_client/src/scope.dart 51 Scope.sendRequest.<async>
dart:async _SyncCompleter.complete
package:json_rpc_2/src/client.dart 178 Client._handleSingleResponse
package:json_rpc_2/src/client.dart 168 Client._handleResponse
dart:async _StreamController.add
package:json_rpc_2/src/peer.dart 117 Peer.listen.<fn>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
package:coverage/src/collect.dart _getAllCoverage
package:coverage/src/collect.dart 26 collect.<async>
dart:async _SyncCompleter.complete
package:coverage/src/util.dart 39 retry.<async>.<fn>.<async>
dart:async _SyncCompleter.complete
package:coverage/src/vm_service_client.dart 90 VMServiceClient.connect.<async>
===== asynchronous gap ===========================
dart:async _asyncThenWrapperHelper
package:coverage/src/collect.dart collect
http://localhost:52493/collect_coverage.dart 21 main.<async>.<fn>.<async>
===== asynchronous gap ===========================
package:stack_trace Chain.capture
http://localhost:52493/collect_coverage.dart 20 main.<async>
Example of a full build log: https://travis-ci.org/corsac-dart/bootstrap/builds/125995686
coverage package is enabled via pub global activate and the log shows that some packages are not latest version, e.g.:
$ pub global activate coverage
Resolving dependencies... (1.5s)
+ args 0.13.4
+ async 1.10.0
+ collection 1.5.1
+ coverage 0.7.3
+ json_rpc_2 1.2.0 (2.0.1 available)
+ logging 0.11.3
+ path 1.3.9
+ pub_semver 1.2.4
+ source_span 1.2.2
+ stack_trace 1.6.5
+ vm_service_client 0.1.3 (0.2.0 available)
I'm wondering if that's the reason, and if we need an updated version of this package published with latest dependencies?
Version 0.9.0 broke JetBrains coverage collection
With the introduction of --uri parameter over port, JetBrains' Dart plugin can no longer collect the coverage data. In fact it would not even start.
Relevant lines in their repo:
https://github.com/JetBrains/intellij-plugins/blob/fcb89e045f291e5117012ba260182a1142fe75b8/Dart/src/com/jetbrains/lang/dart/coverage/DartCoverageProgramRunner.java#L92
Related JetBrains issue:
https://youtrack.jetbrains.com/issue/WEB-25047
Would it not make sense to deprecate port and host options first and remove them at later time? Give the 3rd parties a bit of time to adjust.
Add test for VMService.getAllocationProfile
Minimise transitive dependencies to a reasonable set
f3dfba4 killed off devtools.dart (hooray!) and moved us over to package:vm_service_client. Unfortunately, it appears this change massively increased the set of dependencies; our deps went from args, logging, and path, to 36 deps including analyzer, csslib , html, shelf, and others.
The bad news is that this makes the latest version of this package unacceptable for internal use. With the increased dependency footprint comes a very significant maintenance cost and risk. Since Blaze depends on coverage, every change to csslib, html, or plugin becomes a Blaze-affecting change, which transitively affects the entire codebase. Due to that risk, Blaze-affecting changes are held to a much higher bar than others; testing, review and deployment follows a much more onerous process.
The current situation is that we're locked to an earlier version of coverage internally and I'm maintaining that working with dev-channel SDKs as they're rolled out. I'd really like to get us unforked. As an example of user impact, #107 was filed over a month ago, an internal fix was landed two days later, but it can't be pushed externally due the significant differences in the codebases.
Moving to a vm service package is definitely the right thing to do, but replacing less than 500 loc of devtools.dart with tens of thousands of lines of analyzer, css and html manipulation, etc. is a no-go.
Support for local imports with package
If the user is using package:// in their tests, coverage will not be reported because the "lib" is not appended to the path. Here's a dirty hack to get this working:
pub.bat run coverage:format_coverage --in coverage.json --package-root=/c/Users/codefu/dart/ --report-on=lib | lessIn lib/resolver.dart:
var resolved = resolveSymbolicLinks(p.join(packageRoot, path));
if (resolved == null && path.contains('/')) {
// try adding lib for projects that use package:lib
path = path.replaceFirst('/', '/lib/');
resolved = resolveSymbolicLinks(p.join(packageRoot, path));
}
return resolved;Output as expected!
2| ArdProto(this.port, {this.debug: false}) {
2| _startReading();
| }
If you are fine with this, I can make a pull request.
Add unit tests
Unable to run Dartium coverage on OS X?
I am maybe missing something but it appears that you can't actually run Dartium coverage from OS X. Homebrew-installed dartium is actually just a bash script:
#!/bin/bash
open "/usr/local/Cellar/dartium/1.8.0/Chromium.app" "$@"so the flag --remote-debugging-port=NNNN is not passed in. Trying to open the executable directly, the flag doesn't seem to have any effect:
$ /usr/local/Cellar/dartium/1.8.0/Chromium.app/Contents/MacOS/Chromium --remote-debugging-port=8050Is there some other way to run coverage thru Dartium? The majority of this project's tests depend on dart:html so we can't run coverage without running in Dartium.
Add lcov manipulation API
Specifically the ability to merge lcov.
Get ahead of breaking change to service protcol
https://groups.google.com/a/dartlang.org/d/msg/announce/VxSw-V5tx8k/wPV0GfX7BwAJ
Not sure what's needed – but would be great to validate
Extract functionality from format_coverage.dart to library code
Cannot use coverage and analyzer at the same time
They have incompatible constraints on package_config.
From coverage: package_config: ^1.0.0
From analyzer: package_config: ^0.1.5
Can you loosen your constraints to >=0.1.5 <2.0.0 ?
Would be nice to be able to ignore lines from coverage
I'm not sure how one best would implement this (or if it's even possible), but there are certain lines of code in dart which are intentionally unreachable, an example of such is a private constructor.
class Colors {
Colors._();
}
Which is a pattern used to make a class non-constructable. It's silly that coverage tells me that this line is uncovered:
https://coveralls.io/builds/9847810/source?filename=lib%2Fsrc%2Fmaterial%2Fcolors.dart
But I'm aware of no way with package:coverage to avoid that?
I could imagine a comment syntax, or maybe just teaching package:coverage about this specific idiom? Not sure.
This is not a high priority, but would be nice to have for projects trying for high line or file coverage numbers.
update README.md to explain how to use `pub global` to run the script
Isn't this a perfect candidate to use the new pub global feature to run the script?
pub global activate coverage
pub global run coverage:collect_coverage --port=NNNN -o coverage.json --resume-isolates
I assume pub global has not yet landed in stable, but can't be long ;-)
Single command to generate coverage
Right now, it's a two step process to spin up a VM, then run a separate command, then convert the output.
Can we make it a single command?
Generate coveralls output
Docs here: https://coveralls.io/docs/api
Unhandled exception
Running with git bash 1.9.0, windows 7 64bit, dartvm 1.3.3
One window, start collection:
dart ../coverage/bin/collect_coverage.dart --port=5555 -o coverage.json --resume-isolates
Another window, run tests:
dart --enable-vm-service:5555 --pause_isolates_on_exit test/some_test.dart
Happens majority of the time, but I did have one run succeed.
Uncaught Error: HttpException: Connection closed before full header was received, uri = http://127.0.0.1:5555/isolates/7
114/resume
Unhandled exception:
HttpException: Connection closed before full header was received, uri = http://127.0.0.1:5555/isolates/7114/resume
#0 _rootHandleUncaughtError.. (dart:async/zone.dart:713)
#1 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:23)
#2 _asyncRunCallback (dart:async/schedule_microtask.dart:32)
#3 _asyncRunCallback (dart:async/schedule_microtask.dart:36)
#4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:119)
Explain how coverage statistics are gathered.
It would be nice if it were explained how coverage statistics were gathered. As far as I know, observatory plays some part in it, but that's about it. Ultimately, I'm trying to figure out why my package is showing zero test coverage. Knowing how this package works would be a good start to figuring that out.
Anyhow, I have an issue on my tracker here bgourlie/noBS#3 if anyone would like to take a look at my specific setup.
format_coverage fails on coverage generated from tests run in dartium
I've been able to successfully generate, format, and view coverage of tests run on the VM, but with browser tests it fails for me on the format step:
pub global run coverage:format_coverage --package-root=packages -i browser_coverage.json --lcov > browser_coverage.info
Unhandled exception:
Uncaught Error: FileSystemException: Cannot resolve symbolic links, path = '/Volumes/data/b/build/slave/dartium-mac-full-stable/build/src/dart/sdk/lib/html/html_common/css_class_set.dart' (OS Error: No such file or directory, errno = 2)
Stack Trace:
#0 FileSystemEntity._throwIfError (dart:io/file_system_entity.dart:671)
#1 FileSystemEntity.resolveSymbolicLinksSync (dart:io/file_system_entity.dart:338)
#2 Resolver.resolveSymbolicLinks (package:coverage/src/resolver.dart:66)
#3 Resolver.resolve (package:coverage/src/resolver.dart:55)
#4 LcovFormatter.format.<anonymous closure> (package:coverage/src/formatter.dart:21)
#5 Future.forEach.<anonymous closure>.<anonymous closure> (dart:async/future.dart:336)
#6 Future.Future.sync (dart:async/future.dart:168)
#7 Future.forEach.<anonymous closure> (dart:async/future.dart:336)
#8 Future.Future.sync (dart:async/future.dart:168)
#9 Future.doWhile.<anonymous closure> (dart:async/future.dart:361)
#10 _RootZone.runUnaryGuarded (dart:async/zone.dart:1093)
#11 _RootZone.bindUnaryCallback.<anonymous closure> (dart:async/zone.dart:1122)
#12 _RootZone.runUnary (dart:async/zone.dart:1155)
#13 _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:484)
#14 _Future._propagateToListeners (dart:async/future_impl.dart:567)
#15 _Future._completeWithValue (dart:async/future_impl.dart:358)
#16 _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:412)
#17 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#18 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#19 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#20 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:143)
#0 _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:886)
#1 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#2 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#3 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:143)
Additionally, I was curious if there's a way to specify which files you actually care about for coverage? Or at least a way to exclude certain files. The test coverage I was able to generate/format/view includes coverage for tests and package files, but I'd rather just have the source files.
Null ref exception collecting coverage after upgrading from 0.5.0 to 0.6.0
After upgrading from 0.5.0 to 0.6.0, I get the following error when starting collect_coverage and firing up Dartium (the error doesn't occur until Dartium loads).
Unhandled exception:
Uncaught Error: The null object does not have a method 'map'.
NoSuchMethodError: method not found: 'map'
Receiver: null
Arguments: [Closure: (dynamic) => dynamic]
Stack Trace:
#0 Object.noSuchMethod (dart:core-patch/object_patch.dart:45)
#1 VM.VM.fromJson (package:coverage/src/devtools.dart:109)
#2 VMService.getVM.<anonymous closure> (package:coverage/src/devtools.dart:21)
#3 _RootZone.runUnary (dart:async/zone.dart:1155)
#4 _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:484)
#5 _Future._propagateToListeners (dart:async/future_impl.dart:567)
#6 _Future._completeWithValue (dart:async/future_impl.dart:358)
#7 _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:412)
#8 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#9 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#10 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:84)
#11 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:131)
#0 _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:886)
#1 _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#2 _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#3 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:84)
#4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:131)
I'm starting it like this:
$testInstance = "http://dart-tests/"
$dartiumPath = "C:\Work\Utilities\dartium\chrome.exe"
$dartiumProfilePath = "C:\Work\Utilities\dartium-profile"
$timeoutSeconds = 10
$debuggerPort = 8091
$dartSDK = "%DART_SDK%"
# Start listening for coverage
Write-Host "Starting collect_coverage..."
$proc = Start-Process pub -ArgumentList ("global", "run", "coverage:collect_coverage", "--port=$debuggerPort", "-o coverage/coverage.json", "--connect-timeout=$timeoutSeconds") -PassThru
# Start Dartium and run the tests
Write-Host "Starting Dartium to run tests..."
$dartium = Start-Process $dartiumPath -ArgumentList ("--remote-debugging-port=$debuggerPort", "--user-data-dir=$dartiumProfilePath", "--disable-web-security", $testInstance) -PassThruI'm on Windows 8.1 Enterprise, Dart VM version: 1.8.5 (Tue Jan 13 13:38:34 2015) on "windows_x64".
Dartium version is 38.0.2125.0 (Developer Build 292384).
Migrate to async/await
Now that the SDK restriction is >= 1.9.0, migrate existing code to use async await.
Improve error message around --resume-isolates flag
When collecting coverage with the --resume-isolates flag, you will receive a JsonRpcError: VM must be paused (101) error if the isolates are not paused. This error message could be improved to suggest removing the --resume-isolates flag.
Additionally, the README could include a note or possibly update the recommended usage to explain when the --resume-isolates flag would be needed. As it's written now, the --resume-isolates flag is included in the example command, so it would be easy for someone to copy and paste it and not realize the issue.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.