flutter integration_test flaky result

thekucays 0
Viewed 24

i'm trying to use flutter's integration_test package
but when i run my script it only shows a text "Test Starting...", move to white screen, then test finished

i don't understand, why the test already finished even when no widgets pumped already in my device

here is my 'adaptor' to flutter driver, test_driver/integration_test_driver.dart

import 'package:integration_test/integration_test_driver.dart';
void main() => integrationDriver();

and here is my test file integration_test/app_test.dart

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:myapp/main.dart' as app;

void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();
  group('end-to-end test', () {
    testWidgets('tap on the floating action button, verify counter',
        (tester) async {
        app.main();
        await tester.pumpAndSettle();

        final Finder buttonMasuk =
            find.byKey(const Key('onboarding_masukkeakun123'));
        expect(buttonMasuk, findsOneWidget);
        await tester.tap(buttonMasuk);
    });
  });
}

and i run it using this command

flutter drive --driver=test_driver/integration_test_driver.dart --target=integration_test/app_test.dart --flavor staging --no-sound-null-safety

i don't get it.. the results are flaky.. sometimes it passed, sometimes it failed (with the exact same code)
in my code above, i'm 100% sure that there is no widget with key onboarding_masukkeakun123


when it failed, it prints something like this in my console 
EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
I/flutter (28965): The following assertion was thrown running a test:
I/flutter (28965): The finder "zero widgets with key [<'onboarding_masukkeakun123'>] (ignoring offstage widgets)" (used
I/flutter (28965): in a call to "tap()") could not find any matching widgets.
I/flutter (28965): 
I/flutter (28965): When the exception was thrown, this was the stack:
I/flutter (28965): #0      WidgetController._getElementPoint (package:flutter_test/src/controller.dart:902:7)
I/flutter (28965): #1      WidgetController.getCenter (package:flutter_test/src/controller.dart:841:12)
I/flutter (28965): #2      WidgetController.tap (package:flutter_test/src/controller.dart:273:18)
I/flutter (28965): #3      main.<anonymous closure>.<anonymous closure>.<anonymous closure> (file:///Users/pinhomemacbook9/Documents/WORK/consumer-app/integration_test/app_test.dart:31:22)
I/flutter (28965): <asynchronous suspension>
I/flutter (28965): #4      restoreFlutterError (file:///Users/pinhomemacbook9/Documents/WORK/consumer-app/integration_test/app_test.dart:8:3)
I/flutter (28965): <asynchronous suspension>
I/flutter (28965): #5      main.<anonymous closure>.<anonymous closure> (file:///Users/pinhomemacbook9/Documents/WORK/consumer-app/integration_test/app_test.dart:23:7)
I/flutter (28965): <asynchronous suspension>
I/flutter (28965): #6      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:176:15)
I/flutter (28965): <asynchronous suspension>
I/flutter (28965): #7      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:817:5)
I/flutter (28965): <asynchronous suspension>
I/flutter (28965): 
I/flutter (28965): The test description was:
I/flutter (28965):   tap on the floating action button, verify counter
I/flutter (28965): ════════════════════════════════════════════════════════════════════════════════════════════════════

when it passed, it prints something like this

I/flutter (27514):   'package:flutter_test/src/binding.dart': Failed assertion: line 785 pos 14: '_pendingExceptionDetails != null': A test overrode FlutterError.onError but either failed to return it to its original state, or had unexpected additional errors
that it could not handle. Typically, this is caused by using expect() before restoring FlutterError.onError.
I/CleverTapResponse(27514): Done processing response!
I/flutter (27514):   dart:core-patch/errors_patch.dart 46:39       _AssertionError._doThrowNew
I/flutter (27514):   dart:core-patch/errors_patch.dart 36:5        _AssertionError._throwNew
I/flutter (27514):   package:flutter_test/src/binding.dart 785:14  TestWidgetsFlutterBinding._runTest.handleUncaughtError
I/flutter (27514):   package:flutter_test/src/binding.dart 790:9   TestWidgetsFlutterBinding._runTest.<fn>
I/flutter (27514):   dart:async/zone.dart 1109:14                  _Zone._processUncaughtError
I/flutter (27514):   dart:async/zone.dart 1312:5                   _CustomZone.handleUncaughtError
I/flutter (27514):   dart:async/future_impl.dart 717:16            Future._propagateToListeners
I/flutter (27514):   dart:async/future_impl.dart 610:5             Future._completeError
I/flutter (27514):   dart:async-patch/async_patch.dart 274:13      _completeOnAsyncError
I/flutter (27514):   package:flutter_test/src/binding.dart         TestWidgetsFlutterBinding._runTestBody
I/flutter (27514):   
I/flutter (27514): Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
I/flutter (27514): For example, 'dart test --chain-stack-traces'.
I/flutter (27514): 00:01 +0 -1: (tearDownAll)
I/flutter (27514): 00:01 +1 -1: Some tests failed.
All tests passed.

i also tried to 'catch' the actual error by adding restoreFlutterError function, and modify my test step a little to something like this, but the results still flaky

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:myapp/main.dart' as app;

Future<void> restoreFlutterError(Future<void> Function() call) async {
  final originalOnError = FlutterError.onError!;
  await call();
  final overriddenOnError = FlutterError.onError!;

  // restore FlutterError.onError
  FlutterError.onError = (FlutterErrorDetails details) {
    if (overriddenOnError != originalOnError) overriddenOnError(details);
    originalOnError(details);
  };
}

void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();
  group('end-to-end test', () {
    testWidgets('tap on the floating action button, verify counter',
        (tester) async {
      await restoreFlutterError(() async {
        app.main();
        await tester.pumpAndSettle();
        final Finder buttonMasuk =
            find.byKey(const Key('onboarding_masukkeakun123'));
        expect(buttonMasuk, findsOneWidget);
        await tester.tap(buttonMasuk);
      });
    });
  });
}

so my questions are:

  • what makes my test flaky? did i missed to call pumpAndSettle() somewhere?
  • whether the test results passed or failed, it only shows "Test starting..." text, move to white screen, then my test finished.. no widgets appeared, so why the test already finished

someone enlighten me please
i'm on flutter 2.5.0 btw

[✓] Flutter (Channel stable, 2.5.0, on macOS 12.4 21F79 darwin-arm, locale en-ID)
    • Flutter version 2.5.0 at /Users/pinhomemacbook9/fvm/versions/2.5.0
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 4cc385b4b8 (1 year ago), 2021-09-07 23:01:49 -0700
    • Engine revision f0826da7ef
    • Dart version 2.14.0
integration-testing flutter-integration-test dart flutter
0 Answers
Related