Coder Social home page Coder Social logo

ibautomater's Introduction

QuantConnect / IBAutomater

IBAutomater is an automation tool for IB Gateway v978 and above. It supports starting, stopping and automated logins for IB Gateway. This package is best used in conjunction with LEAN. LEAN handles the automatic login and portfolio construction once you start the IB Gateway.

Installation

IBAutomater is available on NuGet:

PM> Install-Package QuantConnect.IBAutomater

To get started with IBAutomater, first you will need to get the library itself. The easiest way to do this is to install the package into your project using NuGet. Using Visual Studio this can be done in two ways.

Using the package manager

In Visual Studio right click on your solution and select 'Manage NuGet Packages for solution...'. A screen will appear which initially shows the currently installed packages. At the top select 'Browse'. This will let you download the package from the NuGet server. In the search box type 'QuantConnect.IBAutomater'. The QuantConnect.IBAutomater package should come up in the results. After selecting the package you can then on the right hand side select in which projects in your solution the package should install. After you've selected all projects you wish to install and use QuantConnect.IBAutomater in click 'Install' and the package will be downloaded and added to your selected projects.

Using the package manager console

In Visual Studio in the top menu select 'Tools' -> 'NuGet Package Manager' -> 'Package Manager Console'. This should open up a command line interface. On top of the interface there is a dropdown menu where you can select the Default Project. This is the project that IBAutomater will be installed in. After selecting the correct project type Install-Package QuantConnect.IBAutomater in the command line interface. This should install the latest version of the package in your project.

After doing either of above steps you should now be ready to start using IBAutomater.

Code example

using QuantConnect.IBAutomater;

// Create a new instance of IBAutomater
_ibAutomater = new IBAutomater.IBAutomater(ibDirectory, ibVersion, userName, password, tradingMode, port, exportIbGatewayLogs);

// You can bind to event handlers to receive the output data.
_ibAutomater.OutputDataReceived += OnIbAutomaterOutputDataReceived;

// Gracefully handle errors
_ibAutomater.ErrorDataReceived += OnIbAutomaterErrorDataReceived;

// Get events once the IB Gateway has exited.
_ibAutomater.Exited += OnIbAutomaterExited;

// Get events once the IB Gateway has auto-restarted.
_ibAutomater.Restarted += OnIbAutomaterRestarted;

// Trigger the IB Gateway to start and login with your configured parameters.
_ibAutomater.Start(false);

// Stop IB Gateway with a simple command.
_ibAutomater.Stop();

How it works

IBAutomater has been implemented as two components:

  1. a Java component (.jar file)

    The IBAutomater.jar file is loaded as a Java agent into the IB Gateway process with the following advantages:

    • remove the requirement of installing Java separately as IB Gateway uses a bundled version of the Java runtime
    • avoid dependencies on the IB Gateway jar libraries
    • allow us to handle auto-restarts requiring full authentication only weekly (instead of daily)

    More information on the Java Instrumentation API can be found here

    This component is responsible for interacting with the IB Gateway user interface, performing the following tasks:

    • entering login credentials
    • configuring IB Gateway settings
    • dismissing message and confirmation windows
  2. a C# component (.dll file), .NET assembly built for both NET Standard 2.0 and NET 5.0

    This component takes care of the following tasks:

    • starting and stopping the IB Gateway
    • notifying the client application of any error conditions
    • firing events such as Exited and Restarted

IB Gateway auto-restarts

As the IB Gateway requires a daily shutdown (or restart), IBAutomater always selects auto-restart instead of auto-logoff because it allows users with 2FA enabled to authenticate only once per week instead of each day.

IBAutomater handles the detection of auto-restarts and notifies the client application with events. These are the two events that have to be handled by the client:

  1. Restarted - The IB Gateway was auto-restarted with no authentication required (soft restart), this happens every day at the configured auto-restart time.

For now IBAutomater does not set the auto-restart time (the default time is 11:45 PM, in the system timezone), but this setting might be configurable in future versions.

  1. Exited - The IB Gateway was closed by IBAutomater (because full authentication is required or there was an error) and the client application is responsible for restarting the IB Gateway via the Start() method.
Example auto-restart event handling code
private void OnIbAutomaterExited(object sender, ExitedEventArgs e)
{
    // check if IB Gateway was closed because of an IBAutomater error
    var result = _ibAutomater.GetLastStartResult();
    CheckIbAutomaterError(result);

    if (!result.HasError)
    {
        // IB Gateway was closed by IBAutomater because the auto-restart token expired or it was closed manually (less likely)
        Console.WriteLine("OnIbAutomaterExited(): IB Gateway close detected, restarting IBAutomater in 10 seconds...");

        // Wait a few seconds for IB Gateway to shutdown
        Thread.Sleep(TimeSpan.FromSeconds(10));

        try
        {
            // Close the client API connection
            Disconnect();

            // Restart IB Gateway
            CheckIbAutomaterError(_ibAutomater.Start(false));

            // Open the client API connection
            Connect();
        }
        catch (Exception exception)
        {
            Console.WriteLine($"OnIbAutomaterExited(): IBAutomaterRestartError - {exception}");
        }
    }
}

private void OnIbAutomaterRestarted(object sender, EventArgs e)
{
    // check if IB Gateway was closed because of an IBAutomater error
    var result = _ibAutomater.GetLastStartResult();
    CheckIbAutomaterError(result);

    if (!result.HasError)
    {
        // IB Gateway was restarted automatically
        Console.WriteLine("OnIbAutomaterRestarted(): IB Gateway restart detected, reconnecting...");

        try
        {
            // Close the client API connection
            Disconnect();

            // Open the client API connection
            Connect();
        }
        catch (Exception exception)
        {
            Console.WriteLine($"OnIbAutomaterRestarted(): IBAutomaterAutoRestartError - {exception}");
        }
    }
}

private void CheckIbAutomaterError(StartResult result)
{
    if (result.HasError)
    {
        // notify the user that an IBAutomater error has occurred
        Console.WriteLine($"CheckIbAutomaterError(): {result.ErrorCode} - {result.ErrorMessage}");
    }
}

IB weekend/maintenance periods

According to the IB documentation, the IB weekend reset times should start on Friday at 23:00 ET and should last four hours, but sometimes the maintenance lasts more and during this period the IB Gateway is unable to complete a Login operation.

For this reason IBAutomater detects failed login attempts during this period and automatically closes the IBGateway application (although IBGateway will sometimes exit by itself).

If the client application intends to continue running through the weekend, it should handle the OnExited event and delay further connection attempts until the next Sunday at 4:00 PM ET (one hour before Forex markets open), when we expect to be able to login successfully.

Two-factor authentication (2FA)

The only 2FA method supported by IBAutomater is the IBKR mobile application with seamless authentication enabled.

The 2FA request will only be sent to the phone when logging in for the first time at startup and once per week after the weekly auto-restart.

When this method is selected, IB Gateway shows a 2FA popup window (after the user/password credentials have been validated) and waits for the user to complete the authentication by entering the PIN on the phone.

If the PIN is correct, the popup window will automatically close and both IB Gateway and IBAutomater will proceed normally.

If the PIN is incorrect (or the user does not send a PIN within 3 minutes) the window will automatically close/reopen and IB Gateway will send another 2FA request to the phone.

After three failed attempts, IBAutomater will be terminated and show the following error message:

The two factor authentication request timed out. 
The request must be confirmed within 3 minutes.

How to build

To build the NuGet package we need to complete the following three steps in order:

  1. Build the Java project
  2. Build the C# solution
  3. Upload the new package to NuGet

1. Java build (with NetBeans)

  • Download Apache NetBeans 12.x here
  • Install Apache NetBeans 12.1 or higher (currently bundled Java version: JDK 15)
  • Open the project in /IBAutomater/java/IBAutomater
  • Build the project: MainMenu -> Run -> Clean and Build Project
    • output: /IBAutomater/java/IBAutomater/dist/IBAutomater.jar

2. C# build (with Visual Studio)

  • Increment the version number in QuantConnect.IBAutomater.csproj
  • Open the solution file: /IBAutomater/IBAutomater.sln
  • Rebuild solution & test locally

3. NuGet upload

  • In the /IBAutomater/ folder, run the following commands:
  dotnet pack QuantConnect.IBAutomater/QuantConnect.IBAutomater.csproj --output ./packages -p:PackageVersion=2.0.XXXX --configuration Release --include-symbols
  • Extract QuantConnect.IBAutomater.2.0.XXXX.nupkg to sanity test it's contents before pushing
  • Make sure 'IBAutomater.sh' line endings are unix
  dotnet nuget push ./packages/QuantConnect.IBAutomater.2.0.XXXX.nupkg --api-key <API KEY> --source https://api.nuget.org/v3/index.json

ibautomater's People

Contributors

imphasing avatar jaredbroad avatar jhonabreul avatar martin-molinero avatar stefanoraggi avatar tuan-h avatar vslee avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ibautomater's Issues

Upgrade to Use TWS 978

IBAutomater is point to TWS 974 and we might need to upgrade to TWS 978 because of CBOE ONE datafeed (see TWS 978 Release).

I have sent an email to Interactive Brokers asking whether this change is required since we use the latest IB Gateway version (10.19).

Create a complete hello world demo using the code.

A hello-world demonstration is needed for an application which might be connecting and using the IBAutomater.

  • Mock config
  • Create class, bind event handlers.
  • Start IBGateway, Login.
  • Print out logs to console. demo.exe
  • Wait 30 seconds. Stop.

Configurable Sunday Restart

IBAutomater is set to try to connect to IB Gateway as early as Sunday 2 AM EST: IBAutomater.cs#L769
Some users are not awake by that time and will miss the 2FA request.

This restart could be configurable to attend to the user's timezone.

IBAutomater should test for already running Gateway or TWS before trying to start a new Gateway

Most likely for myself and I would assume many others, they already have an IB API process running and don't need a new one started and logged into. Can IBAutomater detect, based on a port given, the directory provided and/or a process check. If there is one running, then connect to that one and don't start new.

The check could be done in Lean but as this is an automation tool I would assume the decision could be handled in here.

IB gateway still restarts after #18

Hi guys,

Yesterday I have tested the changes in the IBAutomater after you merged #18 I see the issues that prevent me from running IBGateway with IBAutomater - gateway restarts. The gateway application goes down on the scheduled logoff/restart time even despite IBAutomater trying to change the restart time in the dialog.

Platform: Win 2012 (x64)

Steps to repro:

  1. Schedule the IB gateway for the time T+3 hours (it did not work for me if I set T+1 hours).
  2. Run IBAutomater.
  3. In three hours IB gateway goes down without recovery.
    I have tried to reproduce this behavior by manually starting IB gateway, scheduling restart for T+3 hours, and trying to keep it alive, but it just restarts.

Any ideas on how to fix this?

Wondering if it is possible to download prev versions of IB Gateway as a temporary solution for now?

Best regards,
Pavel

Docker image

Hello.
Thanks for awesome tool. I've looked into sources, and i understood that its a java program wrapper. I dont know C# and Java tools, and looking to do docker image for that wrapper.I need to know how to build this wrapper to work in docker image in headless mode, if thats possible.
Can you help me?

Multi-targeting (NET Standard 2.0 + NET Framework 4.5)

Currently, IBAutomater only targets NET Framework 4.5. Would it be possible to support multi-targeting, so that it can be used from NET Standard projects as well?

  1. The first step is that the .csproj would need to be modified to use be a SDK-style project instead of a MSBuild one. (PR #16 merged)
  2. After that's done, then the .csproj can be changed from targeting net452 to netstandard20

Add handler for "Enter security code" popup window

Currently, this popup window is not detected.

We should detect it via the "Enter security code" window title, add a window handler for it and log an appropriate message so client applications can be notified of this error condition.

Trading Mode toggle button not found

Hi,

I am running into the issue that it seems the automator cannot locate the switch for trading mode. The log is shown as following. Does anyone have similar experience before?

Thanks,
R

StartIBGateway(): starting IBGateway
StartIBGateway(): IBGateway started
Window event: [WINDOW_ACTIVATED] - Window title: [IB Gateway]
Window event: [WINDOW_OPENED] - Window title: [IB Gateway]
Error: java.lang.Exception: Trading Mode toggle button not found

Unresponsive IB Gateway in Digital Ocean

This GitHub Issue is created in Lean-CLI and IBAutomater because it has not been observed in QuantConnect Cloud or locally, only within Digital Ocean environment.

  1. The algorithm crashes with a "no connected" message. the crash is raised, in general, at 5.00 AM UTC.
  2. The algorithm is unable to connect with another paper-trading account. IB Gateway gets frozen for 15 mins and crashes because it is not able to connect.

Based on the syslog, it looks like the IBGateway process is still running but not responding, because, after the countdown popups, no other events are received (especially the usual Shutdown/OnExited message, which triggers the restart).

We can see the countdown starting after 2021-09-27T23:41:01.1807775Z:

2021-09-27T23:41:01.1924755Z TRACE:: InteractiveBrokersBrokerage.OnIbAutomaterOutputDataReceived(): Content: Set Auto Restart Time (HH:MM) The application will automatically restart in 4 minutes 30 seconds EXIT SESSION SETTING

After connected to the docker instance about 23.47 UTC, we confirmed that the java process which contains the IBGateway was running. In fact, the memory consumption of that process on top command was about 35% of the 1GB.

IB Gateway's launcher.log, we can verify that IBGateway did actually restart:

2021-09-29 23:45:22.662 [WR] INFO  [JTS-Main] - ------------------------------- IB GATEWAY RESTART --------------------------------
2021-09-29 23:45:22.702 [WR] INFO  [JTS-Main] - installer version: 2.82, installer type: standalone, install dir: '/root/ibgateway/', Jts dir: '/root/Jts', VM options file: '/root/ibgateway/ibgateway.vmoptions'

Ref.: QuantConnect/lean-cli#34

"Invalid TwsInfo" on IBGateway 978

I am able to log in manually to IBGateway 978.

When I run the demo app, IBGateway pops up but I get an "invalid twsInfo" pop-up dialog in IBGateway. Prior to the pop-up, the IBGateway progress bar seems to make it about halfway. I get the following output to console from IBAutomator:

2020-03-21T20:30:43.0697435Z IBAutomater process started - Id:4564
2020-03-21T20:30:43.2187452Z StartIBGateway(): starting IBGateway
2020-03-21T20:30:43.4786111Z WARNING: An illegal reflective access operation has occurred
2020-03-21T20:30:43.4786111Z WARNING: Illegal reflective access by twslaunch.security.c (file:/C:/Jts/ibgateway/978/jars/twslaunch-978.jar) to field sun.awt.Win32GraphicsDevice.pfDisabled
2020-03-21T20:30:43.4796077Z WARNING: Please consider reporting this to the maintainers of twslaunch.security.c
2020-03-21T20:30:43.4796077Z WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
2020-03-21T20:30:43.4796077Z WARNING: All illegal access operations will be denied in a future release
2020-03-21T20:30:44.5102290Z StartIBGateway(): IBGateway started
2020-03-21T20:30:45.2387904Z Window event: [WINDOW_ACTIVATED] - Window title: [IB Gateway]
2020-03-21T20:30:45.2407906Z Window event: [WINDOW_OPENED] - Window title: [IB Gateway]
2020-03-21T20:30:45.2407906Z Trading mode: paper
2020-03-21T20:30:45.2417912Z Click button: [Paper Log In]
2020-03-21T20:30:45.6268515Z Window event: [WINDOW_DEACTIVATED] - Window title: [IB Gateway]
2020-03-21T20:30:45.6298502Z Window event: [WINDOW_ACTIVATED] - Window title: [Loading...]
2020-03-21T20:30:45.6308522Z Window event: [WINDOW_OPENED] - Window title: [Loading...]
2020-03-21T20:30:45.7268551Z Window event: [WINDOW_DEACTIVATED] - Window title: [Loading...]
2020-03-21T20:30:45.7268551Z Window event: [WINDOW_ACTIVATED] - Window title: [Authenticating (trying for another 19 seconds)...]
2020-03-21T20:30:45.7358517Z Window event: [WINDOW_OPENED] - Window title: [Authenticating (trying for another 19 seconds)...]
2020-03-21T20:30:46.1977042Z Window event: [WINDOW_DEACTIVATED] - Window title: [Requesting startup parameters...]
2020-03-21T20:30:46.1977042Z Window event: [WINDOW_ACTIVATED] - Window title: [null]
2020-03-21T20:30:46.2007044Z Window event: [WINDOW_OPENED] - Window title: [null]
2020-03-21T20:30:46.2007044Z Error: IB API toggle button not found
2020-03-21T20:30:50.2109384Z Window event: [WINDOW_DEACTIVATED] - Window title: [null]
2020-03-21T20:30:53.5881646Z Window event: [WINDOW_ACTIVATED] - Window title: [null]

Support automating automatic logoff in >974 IB Gateway

After version 974 of IB Gateway, IB re-introduced the automatic logoff every day thing that has plagued IB automation with TWS for ages.. now in IB Gateway latest (976 at this time) - It would be great if IBAutomater could support logging in again when this popup appears.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.