kamax-matrix / mxisd

Federated Matrix Identity Server

mxisd - Federated Matrix Identity Server

• Overview
• Features
• Use cases
• Getting Started
• Support
• Contribute
• Powered by mxisd
• FAQ
• Contact

---

This project is no longer maintained as a standalone product and is currently being merged in Gridify Server.
See the End of Life notice for more details.

---

Overview

mxisd is a Federated Matrix Identity server for self-hosted Matrix infrastructures with enhanced features. As an enhanced Identity service, it implements the Identity service API and several extra features that greatly enhance user experience within Matrix. It is the one stop shop for anything regarding Authentication, Directory and Identity management in Matrix built in a single coherent product.

mxisd is specifically designed to connect to an existing on-premise Identity store (AD/Samba/LDAP, SQL Database, Web services/app, etc.) and ease the integration of a Matrix infrastructure within an existing one.
Check our FAQ entry to know if mxisd is a good fit for you.

The core principle of mxisd is to map between Matrix IDs and 3PIDs (Third-Party IDentifiers) for the Homeserver and its users. 3PIDs can be anything that uniquely and globally identify a user, like:

• Email address
• Phone number
• Skype/Live ID
• Twitter handle
• Facebook ID

If you are unfamiliar with the Identity vocabulary and concepts in Matrix, please read this introduction.

Features

Identity: As a regular Matrix Identity service:

• Search for people by 3PID using its own Identity stores (Spec)
• Invite people to rooms by 3PID using its own Identity stores, with notifications to the invitee (Email, SMS, etc.) (Spec)
• Allow users to add/remove 3PIDs to their settings/profile via 3PID sessions (Spec)
• Register accounts on your Homeserver with 3PIDs (Spec)

As an enhanced Identity service:

• Federation: Use a recursive lookup mechanism when searching and inviting people by 3PID, allowing to fetch data from:
  • Own Identity store(s)
  • Federated Identity servers, if applicable to the 3PID
  • Arbitrary Identity servers
  • Central Matrix Identity servers
• Session Control: Extensive control of where 3PIDs are transmitted so they are not leaked publicly by users
• Registration control: Control and restrict user registration based on 3PID patterns or criterias, like a pending invite
• Authentication: Use your Identity stores to perform authentication in synapse via the REST password provider
• Directory search which allows you to search for users within your organisation, even without prior contact within Matrix using arbitrary search terms
• Auto-fill of user profile (Display name, 3PIDs)
• Bridge Integration: Automatically bridge users without a published Matrix ID

Use cases

• Use your existing Identity stores, do not duplicate your users information
• Auto-fill user profiles with relevant information
• As an organisation, stay in control of your data so it is not published to other servers by default where they currently cannot be removed
• Users can directly find each other using whatever attribute is relevant within your Identity store
• Federate your Identity server so you can discover others and/or others can discover you

Also, check our FAQ entry to know if mxisd is a good fit for you.

Getting started

See the dedicated document

Support

Troubleshooting

A basic troubleshooting guide is available here.

Powered by mxisd

The following projects use mxisd under the hood for some or all their features. Check them out!

• matrix-docker-ansible-deploy
• matrix-register-bot

FAQ

See the dedicated document

Contact

Get in touch via:

• Email: see our website: Kamax.io

mxisd's Issues

Web-based admin UI

Provide an admin user interface to control mxisd to perform current system administration, status check and overall control. There is currently nothing, and basic debugging/checks requires doing API calls.

missing bindDn or bindPassword in config should bind anonymously

My infrastructure LDAP is allowing limited anonymous queries, so I just commented out the bindDn and bindPassword fields from the mxisd.yaml config file :

ldap:
  enabled: true
  tls: false
  host: '10.1.2.3'
  port: 389
  # bindDn: 'cn=user,dc=domain,dc=com'
  # bindPassword: 'notthatone'

mxisd starts correctly, but the first request to validate anything returns an exception (to the client):

{"timestamp":1504707331374,"status":500,"error":"Internal Server Error","exception":"groovy.lang.GroovyRuntimeException","message":"Ambiguous method overloading for method org.apache.directory.ldap.client.api.LdapNetworkConnection#bind.\nCannot resolve which method to invoke for [null, null] due to overlapping prototypes between:\n\t[class java.lang.String, class java.lang.String]\n\t[class org.apache.directory.api.ldap.model.name.Dn, class java.lang.String]","path":"/_matrix/identity/api/v1/lookup"}

Setting the fields to the empty string '' binds anonymously.

It would be good to set them to '' if they are null to avoid this.

Support well-known URLs for recursive lookups

While doing a recursive lookups and contacting another identity server, support well-known URLs to discover where the service is located.

This provider would be the preferred remote implementation over all others, and would come just after the local connectors.
At time of writing, this would be after LDAP lookup and before the DNS SRV lookup.

See RFC 5785 for reference.

Unable to add email address

While Adding a mail address to my account getting error as follows

{"timestamp":1501833776376,"status":501,"error":"Not Implemented","exception":"io.kamax.mxisd.exception.NotImplementedException","message":"No message available","path":"/_matrix/identity/api/v1/validate/email/requestToken"} (Status 501)
OK

Directory Search problem with Rest backend

Expected Behavior

Im configuring a mxisd server in Remote sessions only and has only one REST backend.
and i want to use the Directory Search feature.
if everything ok, i think my Rest directory search endpoint server will receive the search request.

config file:

rest:
  enabled: true
  host: 'http://restbackendip:port'
  endpoints:
    auth: '/_mxisd/backend/api/v1/auth/login'
    directory: '/_mxisd/backend/api/v1/directory/user/search'
    identity:
      single: '/_mxisd/backend/api/v1/identity/lookup/single'
      bulk: '/_mxisd/backend/api/v1/identity/lookup/bulk'

session:
  policy:
    validation:
      enabled: true
      forLocal:
        enabled: true
        toLocal: false
        toRemote:
          enabled: true
      forRemote:
        enabled: true
        toLocal: false
        toRemote:
          enabled: true

Actual Behavior

The entrypoint got nothing. I noticed that no Directory provider was loaded when the program initing.

2017-12-23 21:57:55.759  INFO [           main]    i.k.m.directory.DirectoryManager : Directory providers:
2017-12-23 21:57:56.995  INFO [           main] k.m.MatrixIdentityServerApplication : Started MatrixIdentityServerApplication in 7.358 seconds (JVM running for 9.061)

but the providers will be load correctly if i use ldap or sql backend.

I found that there are no @component and @Autowired in RestDirectoryProvider class unlike the LdapDirectoryProvider or the sql one. Is the intentionally? what if i add them?

Steps to Reproduce the Problem

1. do the session and rest configuration
2. start mxisd server

Specifications

• Version: master branch 6a5a4b3
• Platform: ubuntu 16.04
• Subsystem:

Enable directory search by sql

I am trying to activate directory search as documented in directory-users.md

I see that the query passes one param that should be i believe the seach query? but when I see in the database :

Query:
select * from user_directory;
Response:
userId: @username:idigid.com
displayName: NULL

now this user just signed up so no display name is present but is this the right behavior? shouldn't i be able to search the user with "username" and not needing to write "@username:idigid.com"

Group/Community integrations into Directory lookup and Invitations

To fill the current gap with groups not being able to be used for invitations to rooms:

• Integrate groups when doing a search to perform invite, adding them to the Directory feature, possibly via a special namespace.
• Resolve those special/virtual users so groups invite can be turned into user invites.

At this point, groups means both Identity store groups (if supported) and native Matrix communities (with proper integration).

LDAP Lookup - Support list of LDAP servers

Hey,
as discussed earlier, I'd like you to consider to have the ldap: section in the config to be a list just as the "password_providers:" section in matrix-synapse-ldap3 is currently (probably by coincidence).

This will enable us to
a) query more than one ldap (domain) for a potential hit (in the order of your entry to the list item)
or
b) query more than one e.g. Active Directory/Samba controller (in the order of your entry). Of course for this you could just use the ad domain like domain.local and DNS will return (round robin) one controller, but it does not help if that server would be done (HA).

Currently there is no further handling in matrix-synapse-ldap3, it will stop with the first correct(!) hit as it's just a simple for each in the password_auth_providers[1] module. While this seems not optimal I think this behavior is acceptable on an auth-provider as the chance that two dirs have the same e.g. email and matching password/bind is probably nil and wouldnt change the effect that the user get's authenticated properly. I do not know if there would be a better way or even a need to differently in mxisd?
Best result would be to e.g. on invite to new room members show all hits as a list of |email | |uid| in supporting clients. Just use Riot and search for e.g. "test" on invite new room members while on matrix.org to see that at least Riot supports that.

Let me know if I need to be more specific.
Thx, cu Mike

[1] https://github.com/matrix-org/synapse/blob/f5a4001bb116c468cc5e8e0ae04a1c570e2cb171/synapse/config/password_auth_providers.py#L35 ,

Invite email does not show the name of sender

Steps to reproduce

1. Invite any registered user by email

Response:

1. Invited user gets an email

In the email the name of the user wh has invited does not not show and instead it says
"None has invited you into a room [Room D] on"

%SENDER_ID% has value but %SENDER_NAME% and %SENDER_NAME_OR_ID% have "None" as their values

cat src/main/resources/threepids/email/invite-template.eml

Subject: You have been invited to %DOMAIN%
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="7REaIwWQCioQ6NaBlAQlg8ztbUQj6PKJ"

--7REaIwWQCioQ6NaBlAQlg8ztbUQj6PKJ
Content-Type: text/plain; charset=UTF-8
Content-Disposition: inline

Hi,

%SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on
Riot. To join the conversation, register an account on http://%DOMAIN%

You can also register an account on a public server, like vastri.in, by going to
https://vastri.in/#/register?%INVITE_MEDIUM%=%INVITE_ADDRESS%


About OnePinger:

Riot is an open standard for interoperable, decentralised, real-time communication
over IP, supporting group chat, file transfer, voice and video calling, integrations to
other apps, bridges to other communication systems and much more. It can be used to power
Instant Messaging, VoIP/WebRTC signalling, Internet of Things communication.

Thanks,

%DOMAIN_PRETTY% Admins

--7REaIwWQCioQ6NaBlAQlg8ztbUQj6PKJ
Content-Type: multipart/related;
	boundary="M3yzHl5YZehm9v4bAM8sKEdcOoVnRnKR";
	type="text/html"

--M3yzHl5YZehm9v4bAM8sKEdcOoVnRnKR
Content-Type: text/html; charset=UTF-8
Content-Disposition: inline

<!doctype html>
<html lang="en">
    <head>
        <style type="text/css">
body {
    margin: 0px;
}

pre, code {
    word-break: break-word;
    white-space: pre-wrap;
}

#page {
    font-family: 'Open Sans', Helvetica, Arial, Sans-Serif;
    font-color: #454545;
    font-size: 12pt;
    width: 100%%;
    padding: 20px;
}

#inner {
    width: 640px;
}
        </style>
    </head>
    <body>
        <table id="page">
            <tr>
                <td> </td>
                <td id="inner">
<p>Hi,</p>
test:: %SENDER_ID% %SENDER_NAME%

<p>%SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on
OnePinger. To join the conversation, register an account on <a href="http://%DOMAIN%">%DOMAIN%</a>.</p>

<p>You can also register an account on a public server, like vastri.in, by following
<a href="https://vastri.in/#/register?%INVITE_MEDIUM%=%INVITE_ADDRESS%">this link</a>.</p>

<br>
<p>About OnePinger:</p>

<p>OnePinger is an open standard for interoperable, decentralised, real-time communication
   over IP, supporting group chat, file transfer, voice and video calling, integrations to
   other apps, bridges to other communication systems and much more. It can be used to power
   Instant Messaging, VoIP/WebRTC signalling, Internet of Things communication.</p>

<p>Thanks,</p>

<p>%DOMAIN_PRETTY% Admins</p>
                </td>
                <td> </td>
            </tr>
        </table>
    </body>
</html>
--M3yzHl5YZehm9v4bAM8sKEdcOoVnRnKR--

--7REaIwWQCioQ6NaBlAQlg8ztbUQj6PKJ--

Support phone numbers 3PID

Handle mapping management

Generic issue to track the implementation of 3PID sessions and 3PID bind.

• Framework to handle 3PID binds (#23)
• Bind management for the local Identity server
  • Email (#31)
  • Phone number
• Bind management for proxying to a remote Identity server
  • Email (#31)
  • Phone number

msisdn local validation doesn't work

Hello! I'm using local sessions only config and set up synapseSql config section to my postgresql backend.

When I add email it works pretty well. Only one verification email was sent with link to my own matrix.mobilap.ru server.

Mar 08 09:03:58 matrix mxisd[15850]: .986  INFO [nio-8090-exec-1]  i.k.m.c.i.v1.SessionRestController : Request POST: http://matrix.mobilap.ru/_matrix/identity/api/v1/validate/email/requestToken
Mar 08 09:03:59 matrix mxisd[15850]: .016  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : Server 172.17.0.5 is asking to create session for email:[email protected] (Attempt #1) - Next link: null
Mar 08 09:03:59 matrix mxisd[15850]: .038  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : No existing session for email:[email protected]
Mar 08 09:03:59 matrix mxisd[15850]: .043  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : Is 3PID bound to local domain? false
Mar 08 09:03:59 matrix mxisd[15850]: .054  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : Generated new session 1520499839044 to validate email:[email protected] from server 172.17.0.5
Mar 08 09:03:59 matrix mxisd[15850]: .055  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : Session 1520499839044 for email:[email protected]: sending local validation notification
Mar 08 09:03:59 matrix mxisd[15850]: .059  INFO [nio-8090-exec-1] enericTemplateNotificationGenerator : Generating notification content for 3PID Session validation
Mar 08 09:03:59 matrix mxisd[15850]: .202  INFO [nio-8090-exec-1]  i.k.m.t.c.email.EmailSmtpConnector : Sending invite to [email protected] via SMTP using email-smtp.eu-west-1.amazonaws.com:25
Mar 08 09:03:59 matrix mxisd[15850]: .229  INFO [nio-8090-exec-1]  i.k.m.t.c.email.EmailSmtpConnector : Connecting to email-smtp.eu-west-1.amazonaws.com:25
Mar 08 09:04:00 matrix mxisd[15850]: .393  INFO [nio-8090-exec-1]  i.k.m.t.c.email.EmailSmtpConnector : Invite to [email protected] was sent
Mar 08 09:04:00 matrix mxisd[15850]: .421  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : Stored session 1520499839044
Mar 08 09:04:49 matrix mxisd[15850]: .983  INFO [nio-8090-exec-2]      i.k.m.c.i.v1.SessionController : Requested: http://matrix.mobilap.ru/_matrix/identity/api/v1/validate/email/submitToken?sid=1520499839044&client_secret=ooluIjRGw5UpnZaaPySpwYkGfG45N2dB&token=068503
Mar 08 09:04:49 matrix mxisd[15850]: .985  INFO [nio-8090-exec-2]   i.k.mxisd.session.SessionMananger : Attempting validation for session 1520499839044 from 172.17.0.5
Mar 08 09:04:49 matrix mxisd[15850]: .997  INFO [nio-8090-exec-2]   i.k.mxisd.session.SessionMananger : Session 1520499839044 has been validated locally
Mar 08 09:04:50 matrix mxisd[15850]: .002  INFO [nio-8090-exec-2]      i.k.m.c.i.v1.SessionController : Session 1520499839044 was validated
Mar 08 09:04:56 matrix mxisd[15850]: .799  INFO [nio-8090-exec-3]  i.k.m.c.i.v1.SessionRestController : Requested: http://matrix.mobilap.ru/_matrix/identity/api/v1/3pid/getValidated3pid
Mar 08 09:04:56 matrix mxisd[15850]: .864  INFO [nio-8090-exec-4]  i.k.m.c.i.v1.SessionRestController : Requested: http://matrix.mobilap.ru/_matrix/identity/api/v1/3pid/bind
Mar 08 09:04:56 matrix mxisd[15850]: .872  INFO [nio-8090-exec-4]   i.k.mxisd.session.SessionMananger : Session 1520499839044 for email:[email protected]: MXID @drmoriarty:mobilap.ru was bound locally

But when I add my phone number it generates one SMS with verification code, and then client (web riot) asks me for the second verification code. But there aren't any second SMS.
The log:

Mar 08 09:05:35 matrix mxisd[15850]: .924  INFO [nio-8090-exec-5]  i.k.m.c.i.v1.SessionRestController : Request POST: http://matrix.mobilap.ru/_matrix/identity/api/v1/validate/msisdn/requestToken
Mar 08 09:05:35 matrix mxisd[15850]: .956  INFO [nio-8090-exec-5]   i.k.mxisd.session.SessionMananger : Server 172.17.0.5 is asking to create session for msisdn:79602938024 (Attempt #1) - Next link: null
Mar 08 09:05:35 matrix mxisd[15850]: .962  INFO [nio-8090-exec-5]   i.k.mxisd.session.SessionMananger : No existing session for msisdn:79602938024
Mar 08 09:05:35 matrix mxisd[15850]: .964  INFO [nio-8090-exec-5]   i.k.mxisd.session.SessionMananger : Is 3PID bound to local domain? false
Mar 08 09:05:35 matrix mxisd[15850]: .965  INFO [nio-8090-exec-5]   i.k.mxisd.session.SessionMananger : Generated new session 1520499935964 to validate msisdn:79602938024 from server 172.17.0.5
Mar 08 09:05:35 matrix mxisd[15850]: .967  INFO [nio-8090-exec-5]   i.k.mxisd.session.SessionMananger : Session 1520499935964 for msisdn:79602938024: sending local validation notification
Mar 08 09:05:35 matrix mxisd[15850]: .968  INFO [nio-8090-exec-5] enericTemplateNotificationGenerator : Generating notification content for 3PID Session validation
Mar 08 09:05:35 matrix mxisd[15850]: .974  INFO [nio-8090-exec-5] i.k.m.t.c.p.PhoneSmsTwilioConnector : Sending SMS notification from [phone] to [phone] with 27 characters
Mar 08 09:05:37 matrix mxisd[15850]: .193  INFO [nio-8090-exec-5]   i.k.mxisd.session.SessionMananger : Stored session 1520499935964
Mar 08 09:05:56 matrix mxisd[15850]: .870  INFO [nio-8090-exec-6]  i.k.m.c.i.v1.SessionRestController : Requested: http://matrix.mobilap.ru/_matrix/identity/api/v1/validate/msisdn/submitToken
Mar 08 09:05:56 matrix mxisd[15850]: .873  INFO [nio-8090-exec-6]   i.k.mxisd.session.SessionMananger : Attempting validation for session 1520499935964 from 172.17.0.5
Mar 08 09:05:56 matrix mxisd[15850]: .878  INFO [nio-8090-exec-6]   i.k.mxisd.session.SessionMananger : Session 1520499935964 has been validated locally
Mar 08 09:05:56 matrix mxisd[15850]: .880  INFO [nio-8090-exec-6]   i.k.mxisd.session.SessionMananger : Creating remote 3PID session for msisdn:79602938024 with local session [1520499935964] to {}
Mar 08 09:05:56 matrix mxisd[15850]: .888  INFO [nio-8090-exec-6]     i.k.m.c.DefaultExceptionHandler : Request POST http://matrix.mobilap.ru/_matrix/identity/api/v1/validate/msisdn/submitToken - Error M_FORBIDDEN: Validating remote 3PID is not allowed

I think that the message Request POST http://matrix.mobilap.ru/_matrix/identity/api/v1/validate/msisdn/submitToken - Error M_FORBIDDEN is the root of issue, but I have no idea how to fix the mxisd behaviour.
May be I miss some config section for it?

Inaccuracy in instructions

After ./gradlew build there no ./build/libs/mxisd as described in readme. But present ./build/libs/mxisd.jar
Posibly needs fix docs?

Handle mapping - accept mappings without checking, but do not publish anywhere

First step of the overall implementation

Validate LDAP settings at startup

Try to perform a bind at startup if LDAP is enable to ensure settings are correct.

Failure to post 3PID bind to synapse

Using a SRV record that points to another hostname than the configured server_name, publishing a 3PID mapping after sending a 3PID room invitation fails with:

Posting onBind event to https://matrix.domain.tld:8448/_matrix/federation/v1/3pid/onbind
Answer body: {"errcode":"M_UNKNOWN","error":"Third party certificate could not be checked"

synapse version: 0.26
synapse install: .deb

Unable to send emails to multiple domains

Hello

I have an urgent issue

I have kamax configured but email sending is failing with these logs

Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .385  INFO [onPool-worker-0]    i.k.m.matrix.IdentityServerUtils : https://live.in is not a usable Identity Server
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .385  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://matrix.or
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .506  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Empty 3PID mapping from https://matrix.org
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .506  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://vector.im
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Empty 3PID mapping from https://vector.im
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0]  i.k.m.invitation.InvitationManager : No mapping for pending invite @karan:idigid.com:!kRFGZiOEytBlRMxhZ
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0]  i.k.m.invitation.InvitationManager : Searching for mapping created since invite @rohithzr:idigid.com:!r
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0] l.s.RecursivePriorityLookupStrategy : Host Internal allowed for recursion: true
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0]  i.k.m.l.provider.DnsLookupProvider : Performing DNS lookup for [email protected]
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0]  i.k.m.l.provider.DnsLookupProvider : Domain name for [email protected]: live.com
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0]    i.k.m.matrix.IdentityServerUtils : live.com is not an URL, using as-is
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0]    i.k.m.matrix.IdentityServerUtils : Discovery Identity Server for live.com
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0]    i.k.m.matrix.IdentityServerUtils : Performing SRV lookup
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .625  INFO [onPool-worker-0]    i.k.m.matrix.IdentityServerUtils : Lookup name: _matrix-identity._tcp.live.com
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .626  INFO [onPool-worker-0]    i.k.m.matrix.IdentityServerUtils : No SRV record for _matrix-identity._tcp.live.com
Dec 18 11:17:51 ip-172-31-19-34 mxisd[2536]: .626  INFO [onPool-worker-0]    i.k.m.matrix.IdentityServerUtils : Performing basic lookup using domain name live.com
Dec 18 11:17:52 ip-172-31-19-34 mxisd[2536]: .159  INFO [onPool-worker-0]    i.k.m.matrix.IdentityServerUtils : https://live.com is not a usable Identity Server: com.google.gson.
Dec 18 11:17:52 ip-172-31-19-34 mxisd[2536]: .159  INFO [onPool-worker-0]    i.k.m.matrix.IdentityServerUtils : https://live.com is not a usable Identity Server

What can be the issue?

MySQL backend for authentication

I am looking at setting up MySQL backend using synapse+mxisd+mysql. Going through documentation can't seem to find any reference for MySQL settings. It only says type: sqlite or postgresql

Though in application.example it says
###############

SQL Backend

###############

If you would like to integrate with a MySQL/MariaDB/PostgreQL/SQLite DB,

see https://github.com/kamax-io/mxisd/blob/master/docs/backends/sql.md

Do we have any example how to achieve that if supported?

Follow LDAP referrals

Current code does not follow LDAP referrals for simplicity.

This issue represent the work to implement such support, including an config parameter to enable referrals follow or not.

3PID Invitation lookups don't respect recursiveness settings

When performing /store-invite, mxisd makes a lookup for a possible mapping, but recursive mode is hard-coded for ecosystem compatibility reasons.
mxisd allows a full isolated system except for that hard-coded part.

While the rationale behind enforcing recursive lookups for invite still stands (not create an isolated island), it makes little sense to allow it in all the other modules but not in this one.

We should provide an option to disable recursiveness for invites and MUST label it as Danger zone or similar, so people understand the implications of doing so. We should also log warnings about it, so in case of debug/support, we are aware of it.

Add valid use cases for mxisd on the README

mxisd use cases are not always obvious, especially given the fog/instability surrounding Identity in the Matrix spec/ecosystem.

To ensure users understand the purpose of mxisd and an Identity Server in general, valid use cases should be documented.

User Directory search Sending Invites

Hi,

first thanks for this great job it's helping us a lot.

we have a problem that when we use the User Directory (LDAP Backend) if we opened an invite with a user that doesn't have Riot installed we would like an Email to be sent to him as an invite can you please help us do that.

Mxisd Config

matrix.domain: *******
key.path: /var/mxisd/sign.key
storage.provider.sqlite.database: /var/mxisd/mxisd.db

logging:
  level:
    org:
      springframework: 'WARN'
      apache:
        catalina: 'WARN'
        directory: 'WARN'

lookup:
  recursive:
    enabled: true
    allowedCidr:
      - '0.0.0.0/0'

dns.overwrite.homeserver.client:
  - name: '${matrix.domain}'
    value: 'http://localhost:8008'

ldap.enabled: true
ldap.connection.host: '*******'
ldap.connection.bindDn: 'uid=*******,cn=users,cn=accounts,dc=earthlink,dc=iq'
ldap.connection.bindPassword: '*******'
ldap.connection.baseDn: 'cn=users,cn=accounts,dc=earthlink,dc=iq'
ldap.connection.tls: false
ldap.attribute.uid.type: 'uid'
ldap.attribute.uid.value: 'uid'
ldap.attribute.name: 'cn'
ldap.attribute.threepid.email:
  - 'mail'

ldap.attribute.threepid.msisdn:
  - 'telephoneNumber'

directory.exclude.homeserever: true

both "Mxisd" and the "home server" is behind nginx reverse proxy

Proxy 3pid registration requests to a different Identity server

It would be nice to have the opportunity to configure proxy 3pid assign requests to preffered Identity server (specified in config).

It will be useful when user wants assign additional 3pid not listed in LDAP.

use TLS to connect to LDAP

I tried to connect without TLS, seems working.
I changed the settings to use TLS as i did for synaps, but it didn't work.
Did i miss something? I always get:
"status":500,"error":"Internal Server Error","exception":"org.apache.directory.ldap.client.api.exception.InvalidConnectionException","message":"Cannot connect to the server: Connection refused","path":"/_matrix/identity/api/v1/lookup"}

Can you please give me an example?

Limit remote 3pid lookups

First I'd like to say thank you for your work on this project. The documentation for Sydent is very poor and without this project I definitely wouldn't have gotten a Matrix ID server working nearly as quickly.

This is related to issue #51

There needs to be a configurable limit on the number of attempts/rate of attempts that mxisd sends
for remote 3pid lookups. Currently it sends 3PID lookups to matrix.org/vector.im every minute. The real problem is when there is no mapping for the 3PID. It will continue to spam lookups apparently indefinitely?

As an example, my journalctl output | grep https://matrix.org looks like:

Feb 20 22:18:16 ip-10-0-0-43 mxisd[14305]: .869  INFO [onPool-worker-1] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://matrix.org
Feb 20 22:18:17 ip-10-0-0-43 mxisd[14305]: .559  INFO [onPool-worker-1] k.m.l.p.RemoteIdentityServerFetcher : Empty 3PID mapping from https://matrix.org
Feb 20 22:19:17 ip-10-0-0-43 mxisd[14305]: .820  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://matrix.org
Feb 20 22:19:18 ip-10-0-0-43 mxisd[14305]: .770  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Empty 3PID mapping from https://matrix.org
Feb 20 22:20:18 ip-10-0-0-43 mxisd[14305]: .686  INFO [onPool-worker-1] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://matrix.org
Feb 20 22:20:19 ip-10-0-0-43 mxisd[14305]: .388  INFO [onPool-worker-1] k.m.l.p.RemoteIdentityServerFetcher : Empty 3PID mapping from https://matrix.org
Feb 20 22:21:17 ip-10-0-0-43 mxisd[14305]: .387  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://matrix.org
Feb 20 22:21:18 ip-10-0-0-43 mxisd[14305]: .320  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Empty 3PID mapping from https://matrix.org
Feb 20 22:22:17 ip-10-0-0-43 mxisd[14305]: .362  INFO [onPool-worker-1] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://matrix.org
Feb 20 22:22:18 ip-10-0-0-43 mxisd[14305]: .378  INFO [onPool-worker-1] k.m.l.p.RemoteIdentityServerFetcher : Empty 3PID mapping from https://matrix.org
Feb 20 22:23:16 ip-10-0-0-43 mxisd[14305]: .851  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://matrix.org
Feb 20 22:23:17 ip-10-0-0-43 mxisd[14305]: .768  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Empty 3PID mapping from https://matrix.org
Feb 20 22:24:17 ip-10-0-0-43 mxisd[14305]: .372  INFO [onPool-worker-1] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://matrix.org
Feb 20 22:24:18 ip-10-0-0-43 mxisd[14305]: .310  INFO [onPool-worker-1] k.m.l.p.RemoteIdentityServerFetcher : Empty 3PID mapping from https://matrix.org
Feb 20 22:25:16 ip-10-0-0-43 mxisd[14305]: .772  INFO [onPool-worker-0] k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://matrix.org

I currently work around this problem by running a cron job for a script that periodically cleans out the invite and session database, which looks like this:

#!/bin/bash
# This will clean the mxisd database of all entires
echo 'cron job executing to clean mxisd database'
sudo systemctl stop mxisd
sudo sqlite3 /var/opt/mxisd/mxisd.db <<EOF
DELETE FROM session_3pid;
DELETE FROM invite_3pid;
VACUUM;
EOF
sudo systemctl start mxisd
echo 'database clean completed'

Maybe I have something configured wrong, but I feel like there needs to be a failsafe to keep this from happening. T

Thanks,
Pete

Generate invite emails - (when no match is found 3pid <=> email?)

As discussed in Matrix HQ (#matrix:matrix.org) [1] the IS sends an invite email on matrix.org if there is no matching uid.
Mxisd should try to cover this for own HS as well imho. While with ldap dirs this certainly has another prio as with normal IS, it still makes sense to somehow alert people in ldap that they are beeing invited to a room.
I would go so far and - for the special purpose that mxisd serves - would even consider that every invite, which results in a query to mxisd should maybe generate a simple invitation email... though I am not sure how the IS will be informed that the client hits "invite" after the query...
Up for discussion, maybe there is a better way specially since the whole IS thing is not fixed yet.

Thx, Mike

[1]https://matrix.to/#/!cURbafjkfsMDVwdRDQ:matrix.org/$14906345581992869sDSnN:matrix.org

Review documentation in preparation of v1.0 milestone

Several pages are either out of date or need to be completed. It would be best to have those done for the v1.0 milestone so an announcement can be made and users have the best available documentation.
A full review is needed at this point.

Add a configuration option for the pending invite lookups rate

Related to #52

Does not support directory lookup with Authorization header

When triggering /_matrix/client/r0/user_directory/search where mxisd should handle that request I get the following error:

{
    "errcode":"M_INCOMPLETE_REQUEST",
    "error":"Required String parameter 'access_token' is not present",
    "success":false
}

This lets me assume, that the Authorization header, where riot defaults to right now, is not implemented yet.

Support 3PID authentication

To help Homeservers deal with 3PID authentication in a consistent way, and ensure this is consistent with current mxisd Authentication feature and synapse integration, we want to redirect the login endpoint(s) to mxisd which will:

• Transform the 3PID data and proxy the request to the homeserver using regular Matrix ID credentials.
• Proxy the request untouched if a regular Matrix ID is provided.

This approach will be very similar to the Directory feature.

Ping snowping and kr0mbel on Matrix for testing

forward.servers option should handle network discovery

If the URL protocol is not present, Identity server network discovery should take place

Explain Local Sessions

Hello

Referencing to the guide here local-sessions-only

when adding the given config for sessions. I am getting the following error:

Steps:

1. Register using [email protected]
2. Gets a mail with the link :

https://idigid.com/_matrix/identity/api/v1/validate/email/submitToken?sid=1513801081449&client_secret=F3S....&token=82...

3. The link returns 302 to this link

https://vastri.in/#/register?client_secret=OOFb1...&hs_url=https%3A%2F%2Fidigid.com&is_url=https%3A%2F%2Fidigid.com&session_id=tnyW...

this page now opens the same register page with the error as here
https://imgur.com/a/rAasj

What could be I be doing wrong?

Riot+Synapse+Mxisd+LDAP. Not connecting

Hello!. I can't connecting in Riot through LDAP =( What can i do wrong?

I did:

1. Install Synapse on Ubuntu 16 from debian repo. Install with server name "matrix_server"
2. Install Mxisd from .deb. Rename default conf to mxisd.yaml and set matrix.domain to "matrix_server"
3. Install Authenticator module for synapse (https://github.com/kamax-io/matrix-synapse-rest-auth) and add to homeserver.yaml:
   password_providers:

• module: "rest_auth_provider.RestAuthProvider"
  config:
  endpoint: "http://localhost:8090"

4. Add to mxisd.yaml LDAP:

ldap.enabled: true
ldap.connection.host: 'server.matrix.local'
ldap.connection.bindDn: 'CN=service_user,OU=Users,DC=Matrix,DC=Local'
ldap.connection.bindPassword: '***********'
ldap.connection.baseDn: 'OU=Users,DC=Matrix,DC=Local'
ldap.attribute.uid.type: 'uid'
ldap.attribute.uid.value: 'uid'
ldap.attribute.name: 'cn'

5. Saved. Restart services. matrix-synapse and mxisd started. And listen on 8008 and 8090
6. Trying to connect from Riot:

login: service_user
password: **********
URL homerserver: http://matrix_server:8008
URL of the identity server: http://matrix_server:8090

7. No connecting. Invalid username and/or password. But login and pass are 100% true!
8. In logs:
   homeserver:

2017-11-27 11:41:56,746 - synapse.access.http.8008 - 59 - INFO - POST-14- 192.168.0.2 - 8008 - Received request: POST /_matrix/client/r0/login?
2017-11-27 11:41:56,747 - synapse.rest.client.v1.login - 177 - INFO - POST-14- Got login request with identifier: {'type': 'm.id.user', 'user': 'service_user'}, medium: None, address: None, user: 'service_user'
2017-11-27 11:41:56,781 - synapse.handlers.auth - 479 - WARNING - POST-14- Attempted to login as @service_user:matrix_server but they do not exist
2017-11-27 11:41:56,782 - synapse.http.server - 123 - INFO - POST-14- <SynapseRequest at 0x7fb5e2fa7dd0 method=POST uri=/_matrix/client/r0/login? clientproto=HTTP/1.1 site=8008> SynapseError: 403 - Invalid password

mxisd:

Nov 27 11:41:56 matrix_server mxisd[5224]: .759  INFO [nio-8090-exec-7]       i.k.m.b.ldap.LdapAuthProvider : Performing auth for @service_user:matrix_server
Nov 27 11:41:56 matrix_server mxisd[5224]: .776  INFO [nio-8090-exec-7]       i.k.m.b.ldap.LdapAuthProvider : No match were found for @service_user:matrix_server

P.S. 1
service_user - CN attribute of AD user. CN and samAccountname avaible attributes for this user.
Maybe in mxisd.yaml mistake for me?
P.S. 2
My LDAP - Active Directory

Tell me please what i can do wrong?

Thank you advance!

E-mail address mapper

To ease the adoption of Matrix and ensure there is a consistent naming and mapping with what users have, we should be able to use the e-mail to build a matrix ID, in case the info is not available in another attribute.

This issue is to implement an e-mail mapper.
[email protected] would return the Matrix ID of @john.doe:example.org.

Display a more user-friendly message in case of common LDAP errors

Expected cases:

• network timeout
• network connection refused
• invalid bind credentials

Local-part e-mail address mapper

To ease the adoption of Matrix and ensure there is a consistent naming and mapping with what users have, we should be able to use the e-mail to build a matrix ID, in case the info is not available in another attribute.

This issue is to implement a local-part mapper.
[email protected] would return a uid of john.doe which would be use to build the Matrix ID.
Domain part would use the configured domain in the YAML config.

Registration - Email Validation process throws error but still validates

ok I head over to my riot app and create a new account.
First I get an email titled Linking your Email address to your OnePinger account that takes me to a url on my identity server /_matrix/identity/remote/api/v1/validate/requestToken?
Then I get another email Your Matrix Validation Token where the link takes me to https://matrix.org/_matrix/identity
I verify the account Verification successful! Please return to your Matrix client to continue.
I go back to the previous link where it now says Once the validation was successful with the global server, please follow this link to validate it with us. and if I follow this link I get this error
{"errcode":"M_UNKNOWN","error":"An internal server error occured. If this error persists, please contact support with reference #1513682314924","success":false}

1. I want to generate the token on my domain itself and not matrix.org.
2. I want to eliminate this error.

Please let me know if you need anymore information.
Thanks

Support arbitrary ordered list of Identity stores

Currently, mxisd does not let you add configure the order in which identity stores are used, nor using several instances of an Identity store.

To facilitate failover configurations and/or custom ordering of Identity stores during resolution, it would be ideal that the configuration contains a list of identity stores, identified by their type, and a configuration for each.

Example of possible future configuration:

stores:
  - type: ldap
    config:
       server:
         host: "mainLdapServer"
         ...
  - type: ldap
    config:
      server:
        host: "failoverLdapServer"
  - type: rest
    config:
       ...
  - type: synapseSql
    config:
       ...

unable to invite non-registered users for some email ids

Description

I am trying to invite a user that has no entry in the database. I am able to invite most email ids but I cannot invite users with some email ids like:

In the issue matrix-org/synapse#2740 I think it might be an issue for kamax mxisd and not just synapse, so I have referenced both the issues to each other.

These don't work
[email protected]
[email protected]

These work
[email protected]
[email protected]

The URL that is called

https://myHSandISdomain.com/_matrix/client/r0/rooms/!PCxVrQnBCfwqiHRfVu:idigid.com/invite?access_token=Mtoken

Request Params

id_server | idigid.com
medium | email
address | [email protected]

Response Error (Status:500)

{"errcode":"M_UNKNOWN","error":"Internal server error"}

Logs (homeserver)

2017-12-19 12:00:26,075 - synapse.access.http.8008 - 59 - INFO - POST-7794- 34.214.200.51 - 8008 - Received request: POST /_matrix/client/r0/rooms/!PCxVrQnBCfwqiHRfVu%3Aidigid.com/invite?access_token=<redacted>
2017-12-19 12:00:26,076 - synapse.http.client - 96 - INFO - POST-7794- Sending request GET https://idigid.com/_matrix/identity/api/v1/lookup?medium=email&address=talk2rohithzr%40gmail.com
2017-12-19 12:00:26,145 - twisted - 131 - INFO - - Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7fbb7926a398>, <twisted.internet.endpoints._WrapperEndpoint object at 0x7fbb7911a210>)
2017-12-19 12:00:26,147 - synapse.metrics - 162 - INFO - - Collecting gc 0
2017-12-19 12:00:26,152 - synapse.http.client - 105 - INFO - POST-7794- Received response to  GET https://idigid.com/_matrix/identity/api/v1/lookup?medium=email&address=talk2rohithzr%40gmail.com: 200
2017-12-19 12:00:26,153 - synapse.http.client - 96 - INFO - POST-7794- Sending request POST https://idigid.com/_matrix/identity/api/v1/store-invite
2017-12-19 12:00:26,163 - twisted - 131 - INFO - POST-7794- Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7fbb7926a398>, <twisted.internet.endpoints._WrapperEndpoint object at 0x7fbb7911a210>)
2017-12-19 12:00:26,164 - twisted - 131 - INFO - POST-7794- Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7fbb79345f50>, <twisted.internet.endpoints._WrapperEndpoint object at 0x7fbb7911db50>)
2017-12-19 12:00:26,598 - synapse.http.client - 105 - INFO - POST-7794- Received response to  POST https://idigid.com/_matrix/identity/api/v1/store-invite: 400
2017-12-19 12:00:26,599 - synapse.http.server - 139 - ERROR - POST-7794- Failed handle request synapse.http.server._async_render on <synapse.rest.ClientRestResource object at 0x7fbb89d5c910>: <XForwardedForRequest at 0x7fbb790603b0 method=POST uri=/_matrix/client/r0/rooms/!PCxVrQnBCfwqiHRfVu%3Aidigid.com/invite?access_token=<redacted> clientproto=HTTP/1.0 site=8008>
Traceback (most recent call last):
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/synapse/http/server.py", line 116, in wrapped_request_handler
    yield request_handler(self, request, request_metrics)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/synapse/http/server.py", line 257, in _async_render
    callback_return = yield callback(request, **kwargs)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/synapse/rest/client/v1/room.py", line 603, in on_POST
    txn_id
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/synapse/handlers/room_member.py", line 533, in do_3pid_invite
    txn_id=txn_id
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/synapse/handlers/room_member.py", line 639, in _make_and_store_3pid_invite
    inviter_avatar_url=inviter_avatar_url
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/home/ubuntu/.synapse/local/lib/python2.7/site-packages/synapse/handlers/room_member.py", line 745, in _ask_id_server_for_third_party_invite
    token = data["token"]
KeyError: 'token'
2017-12-19 12:00:26,600 - synapse.access.http.8008 - 91 - INFO - POST-7794- 34.214.200.51 - 8008 - {@ro_zivo:idigid.com} Processed request: 525ms (12ms, 0ms) (0ms/0) 67B 500 "POST /_matrix/client/r0/rooms/!PCxVrQnBCfwqiHRfVu%3Aidigid.com/invite?access_token=<redacted> HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0"
2017-12-19 12:00:26,600 - twisted - 131 - INFO - POST-7794- Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7fbb79345f50>, <twisted.internet.endpoints._WrapperEndpoint object at 0x7fbb7911db50>)

Invalid or empty value for configuration key matrix.domain

Hello! On my Ubuntu i install synapse and mxisd.
In homeserver.yaml:
server_name: "matrix_server"

and mxisd-sample.yaml:

matrix:
domain: 'matrix_server'

But MXISS failure =(

administrator@matrix_server:~$ sudo systemctl status mxisd.service
● mxisd.service - mxisd
   Loaded: loaded (/etc/systemd/system/mxisd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Пт 2017-11-24 11:27:21 MSK; 2min 30s ago
  Process: 3884 ExecStart=/usr/bin/mxisd --spring.config.location=/etc/mxisd/ --spring.config.name=mxisd --spring.profiles.active=systemd (code=exited, status=1/FAILURE)
 Main PID: 3884 (code=exited, status=1/FAILURE)

ноя 24 11:27:21 matrix_server mxisd[3884]: ***************************
ноя 24 11:27:21 matrix_server mxisd[3884]: APPLICATION FAILED TO START
ноя 24 11:27:21 matrix_server mxisd[3884]: ***************************
ноя 24 11:27:21 matrix_server mxisd[3884]: Description:
ноя 24 11:27:21 matrix_server mxisd[3884]: Invalid or empty value for configuration key matrix.domain
ноя 24 11:27:21 matrix_server mxisd[3884]: Action:
ноя 24 11:27:21 matrix_server mxisd[3884]: Double check the key value
ноя 24 11:27:21 matrix_server systemd[1]: mxisd.service: Main process exited, code=exited, status=1/FAILURE
ноя 24 11:27:21 matrix_server systemd[1]: mxisd.service: Unit entered failed state.
ноя 24 11:27:21 matrix_server systemd[1]: mxisd.service: Failed with result 'exit-code'.

Help please. Thanks you advanced!

Getting NullPointerException on signup using riot

I have configured a matrix homeserver (synapse) and am configuring the identity server (mxisd) but I am getting a null pointer exception on the basic settings.

Config

matrix.domain: 'https://idigid.com'
key.path: '/home/ubuntu/mxisd/sign.key'
storage.backend: 'sqlite'
storage.provider.sqlite.database: '/home/ubuntu/mxisd/mxisd.db'
threepid.medium.email.connectors.smtp.host: "email-smtp.us-west-2.amazonaws.com"
threepid.medium.email.connectors.smtp.port: 587
threepid.medium.email.connectors.smtp.tls: 1
threepid.medium.email.connectors.smtp.login: "*redacted*"
threepid.medium.email.connectors.smtp.password: "*redacted*"
threepid.medium.email.identity.from: "[email protected]"

It probably has something to do with

PlaceholderNotificationGenerator.java

Error

ubuntu@ip-172-31-19-34:~/mxisd$ java -jar build/libs/mxisd.jar
2017-11-23 07:49:49.944  INFO [           main] k.m.MatrixIdentityServerApplication : Starting MatrixIdentityServerApplication on ip-172-31-19-34 with PID 29678 (/home/ubuntu/mxisd/build/libs/mxisd.jar started by ubuntu in /home/ubuntu/mxisd)
2017-11-23 07:49:49.956  INFO [           main] k.m.MatrixIdentityServerApplication : No active profile set, falling back to default profiles: default
2017-11-23 07:49:55.399  INFO [           main]      i.k.m.c.rest.RestBackendConfig : --- REST backend config ---
2017-11-23 07:49:55.400  INFO [           main]      i.k.m.c.rest.RestBackendConfig : Enabled: false
2017-11-23 07:49:56.093  INFO [           main]    i.k.mxisd.config.ldap.LdapConfig : --- LDAP Config ---
2017-11-23 07:49:56.093  INFO [           main]    i.k.mxisd.config.ldap.LdapConfig : Enabled: false
2017-11-23 07:49:56.115  INFO [           main]  io.kamax.mxisd.config.MatrixConfig : --- Matrix config ---
2017-11-23 07:49:56.119  INFO [           main]  io.kamax.mxisd.config.MatrixConfig : Domain: https://idigid.com
2017-11-23 07:49:56.119  INFO [           main]  io.kamax.mxisd.config.MatrixConfig : Identity:
2017-11-23 07:49:56.124  INFO [           main]  io.kamax.mxisd.config.MatrixConfig : 	Servers: {"root":["https://matrix.org"]}
2017-11-23 07:49:56.184  INFO [           main]  io.kamax.mxisd.config.ServerConfig : --- Server config ---
2017-11-23 07:49:56.188  INFO [           main]  io.kamax.mxisd.config.ServerConfig : Name: https://idigid.com
2017-11-23 07:49:56.188  INFO [           main]  io.kamax.mxisd.config.ServerConfig : Port: 8090
2017-11-23 07:49:56.188  INFO [           main]  io.kamax.mxisd.config.ServerConfig : Public URL: https://https://idigid.com
2017-11-23 07:49:56.242  INFO [           main]  i.kamax.mxisd.config.sql.SqlConfig : --- Generic SQL Provider config ---
2017-11-23 07:49:56.242  INFO [           main]  i.kamax.mxisd.config.sql.SqlConfig : Enabled: false
2017-11-23 07:49:56.276  INFO [           main]      i.k.m.c.t.n.NotificationConfig : --- Notification config ---
2017-11-23 07:49:56.276  INFO [           main]      i.k.m.c.t.n.NotificationConfig : Handlers:
2017-11-23 07:49:56.302  INFO [           main]        i.k.m.c.t.medium.PhoneConfig : --- Phone config ---
2017-11-23 07:49:56.308  INFO [           main]        i.k.m.c.t.medium.PhoneConfig : Generator: template
2017-11-23 07:49:56.308  INFO [           main]        i.k.m.c.t.medium.PhoneConfig : Connector: twilio
2017-11-23 07:49:56.336  INFO [           main]       i.k.m.c.t.c.PhoneTwilioConfig : --- Phone SMS Twilio connector config ---
2017-11-23 07:49:56.339  INFO [           main]       i.k.m.c.t.c.PhoneTwilioConfig : Account SID:
2017-11-23 07:49:56.339  INFO [           main]       i.k.m.c.t.c.PhoneTwilioConfig : Sender number:
2017-11-23 07:49:56.342  INFO [           main] i.k.m.t.c.p.PhoneSmsTwilioConnector : Twilio API has been initiated
2017-11-23 07:49:56.380  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : --- SMS Generator templates config ---
2017-11-23 07:49:56.381  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : Invite: Built-in (threepids/sms/invite-template.txt)
2017-11-23 07:49:56.381  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : Session validation:
2017-11-23 07:49:56.381  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : 	Local: Built-in (threepids/sms/validate-local-template.txt)
2017-11-23 07:49:56.381  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : 	Remote: Built-in (threepids/sms/validate-remote-template.txt)
2017-11-23 07:49:56.459  INFO [           main]        i.k.m.c.t.medium.EmailConfig : --- E-mail config ---
2017-11-23 07:49:56.460  INFO [           main]        i.k.m.c.t.medium.EmailConfig : From: [email protected]
2017-11-23 07:49:56.461  INFO [           main]        i.k.m.c.t.medium.EmailConfig : Name: Https://idigid.com Identity Server
2017-11-23 07:49:56.461  INFO [           main]        i.k.m.c.t.medium.EmailConfig : Generator: template
2017-11-23 07:49:56.461  INFO [           main]        i.k.m.c.t.medium.EmailConfig : Connector: smtp
2017-11-23 07:49:56.481  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : --- E-mail Generator templates config ---
2017-11-23 07:49:56.482  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : Invite: Built-in (threepids/email/invite-template.eml)
2017-11-23 07:49:56.482  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : Session validation:
2017-11-23 07:49:56.482  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : 	Local: Built-in (threepids/email/validate-local-template.eml)
2017-11-23 07:49:56.483  INFO [           main]     i.k.m.c.t.m.EmailTemplateConfig : 	Remote: Built-in (threepids/email/validate-remote-template.eml)
2017-11-23 07:49:56.519  INFO [           main]         i.k.m.c.t.c.EmailSmtpConfig : --- E-mail SMTP Connector config ---
2017-11-23 07:49:56.520  INFO [           main]         i.k.m.c.t.c.EmailSmtpConfig : Host: email-smtp.us-west-2.amazonaws.com
2017-11-23 07:49:56.520  INFO [           main]         i.k.m.c.t.c.EmailSmtpConfig : Port: 587
2017-11-23 07:49:56.520  INFO [           main]         i.k.m.c.t.c.EmailSmtpConfig : TLS Mode: 1
2017-11-23 07:49:56.520  INFO [           main]         i.k.m.c.t.c.EmailSmtpConfig : Login: *redacted*
2017-11-23 07:49:56.520  INFO [           main]         i.k.m.c.t.c.EmailSmtpConfig : Has password: true
2017-11-23 07:49:56.609  INFO [           main]     i.k.m.c.t.c.EmailSendGridConfig : --- Email SendGrid connector config ---
2017-11-23 07:49:56.610  INFO [           main]     i.k.m.c.t.c.EmailSendGridConfig : API key configured?: false
2017-11-23 07:49:56.612  INFO [           main]     i.k.m.c.t.c.EmailSendGridConfig : Identity: {"from":"","name":""}
2017-11-23 07:49:56.613  INFO [           main]     i.k.m.c.t.c.EmailSendGridConfig : Templates: {"invite":{"subject":"","body":{"text":"","html":""}},"session":{"local":{"body":{}},"remote":{"body":{}}}}
2017-11-23 07:49:56.626  INFO [           main]         i.k.m.n.NotificationManager : Found handler raw for medium msisdn
2017-11-23 07:49:56.626  INFO [           main]         i.k.m.n.NotificationManager : Found handler raw for medium email
2017-11-23 07:49:56.626  INFO [           main]         i.k.m.n.NotificationManager : Found handler sendgrid for medium email
2017-11-23 07:49:56.626  INFO [           main]         i.k.m.n.NotificationManager : --- Notification handler ---
2017-11-23 07:49:56.627  INFO [           main]         i.k.m.n.NotificationManager : 	Handler for msisdn: raw
2017-11-23 07:49:56.627  INFO [           main]         i.k.m.n.NotificationManager : 	Handler for email: sendgrid
2017-11-23 07:49:56.811  INFO [           main]   com.j256.ormlite.table.TableUtils : creating table 'invite_3pid'
2017-11-23 07:49:56.967  INFO [           main]   com.j256.ormlite.table.TableUtils : executed create table statement changed 0 rows: CREATE TABLE IF NOT EXISTS `invite_3pid` (`id` VARCHAR , `token` VARCHAR NOT NULL , `sender` VARCHAR NOT NULL , `medium` VARCHAR NOT NULL , `address` VARCHAR NOT NULL , `roomId` VARCHAR NOT NULL , `properties` VARCHAR , PRIMARY KEY (`id`) )
2017-11-23 07:49:56.978  INFO [           main]   com.j256.ormlite.table.TableUtils : creating table 'session_3pid'
2017-11-23 07:49:56.983  INFO [           main]   com.j256.ormlite.table.TableUtils : executed create table statement changed 0 rows: CREATE TABLE IF NOT EXISTS `session_3pid` (`id` VARCHAR , `creationTime` BIGINT NOT NULL , `server` VARCHAR NOT NULL , `medium` VARCHAR NOT NULL , `address` VARCHAR NOT NULL , `secret` VARCHAR NOT NULL , `attempt` INTEGER NOT NULL , `nextLink` VARCHAR , `token` VARCHAR NOT NULL , `validated` BOOLEAN , `validationTime` BIGINT , `isRemote` BOOLEAN NOT NULL , `remoteServer` VARCHAR , `remoteId` VARCHAR , `remoteSecret` VARCHAR , `remoteAttempt` INTEGER , `isRemoteValidated` BOOLEAN NOT NULL , PRIMARY KEY (`id`) )
2017-11-23 07:49:57.148  INFO [g-Init-Reporter]             com.mchange.v2.log.MLog : MLog clients using slf4j logging.
2017-11-23 07:49:57.238  INFO [           main]    com.mchange.v2.c3p0.C3P0Registry : Initializing c3p0-0.9.5.2 [built 08-December-2015 22:06:04 -0800; debug? true; trace: 10]
2017-11-23 07:49:57.340  INFO [           main]  i.kamax.mxisd.config.sql.SqlConfig : --- Synapse SQL Provider config ---
2017-11-23 07:49:57.340  INFO [           main]  i.kamax.mxisd.config.sql.SqlConfig : Enabled: false
2017-11-23 07:49:57.409  INFO [           main]     i.k.mxisd.config.FirebaseConfig : --- Firebase configuration ---
2017-11-23 07:49:57.412  INFO [           main]     i.k.mxisd.config.FirebaseConfig : Enabled: false
2017-11-23 07:49:57.455  INFO [           main] i.k.m.c.RecursiveLookupBridgeConfig : --- Bridge integration lookups config ---
2017-11-23 07:49:57.460  INFO [           main] i.k.m.c.RecursiveLookupBridgeConfig : Enabled: false
2017-11-23 07:49:57.478  INFO [           main] l.s.RecursivePriorityLookupStrategy : 3PID Provider ForwarderProvider is enabled: true
2017-11-23 07:49:57.478  INFO [           main] l.s.RecursivePriorityLookupStrategy : 3PID Provider DnsLookupProvider is enabled: true
2017-11-23 07:49:57.479  INFO [           main] l.s.RecursivePriorityLookupStrategy : 3PID Provider RestThreePidProvider is enabled: false
2017-11-23 07:49:57.479  INFO [           main] l.s.RecursivePriorityLookupStrategy : 3PID Provider LdapThreePidProvider is enabled: false
2017-11-23 07:49:57.479  INFO [           main] l.s.RecursivePriorityLookupStrategy : 3PID Provider GenericSqlThreePidProvider is enabled: false
2017-11-23 07:49:57.479  INFO [           main] l.s.RecursivePriorityLookupStrategy : 3PID Provider SynapseSqlThreePidProvider is enabled: false
2017-11-23 07:49:57.479  INFO [           main] l.s.RecursivePriorityLookupStrategy : 3PID Provider GoogleFirebaseProvider is enabled: false
2017-11-23 07:49:57.481  INFO [           main] l.s.RecursivePriorityLookupStrategy : Found 2 providers
2017-11-23 07:49:57.488  INFO [           main] l.s.RecursivePriorityLookupStrategy : 	- io.kamax.mxisd.lookup.provider.ForwarderProvider
2017-11-23 07:49:57.488  INFO [           main] l.s.RecursivePriorityLookupStrategy : 	- io.kamax.mxisd.lookup.provider.DnsLookupProvider
2017-11-23 07:49:57.489  INFO [           main] l.s.RecursivePriorityLookupStrategy : Recursive lookup enabled: true
2017-11-23 07:49:57.489  INFO [           main] l.s.RecursivePriorityLookupStrategy : 127.0.0.0/8 is allowed for recursion
2017-11-23 07:49:57.490  INFO [           main] l.s.RecursivePriorityLookupStrategy : 10.0.0.0/8 is allowed for recursion
2017-11-23 07:49:57.491  INFO [           main] l.s.RecursivePriorityLookupStrategy : 172.16.0.0/12 is allowed for recursion
2017-11-23 07:49:57.491  INFO [           main] l.s.RecursivePriorityLookupStrategy : 192.168.0.0/16 is allowed for recursion
2017-11-23 07:49:57.491  INFO [           main] l.s.RecursivePriorityLookupStrategy : ::1/128 is allowed for recursion
2017-11-23 07:49:57.613  INFO [           main]     i.k.m.config.DnsOverwriteConfig : --- DNS Overwrite config ---
2017-11-23 07:49:57.616  INFO [           main]     i.k.m.config.DnsOverwriteConfig : Homeserver:
2017-11-23 07:49:57.620  INFO [           main]     i.k.m.config.DnsOverwriteConfig : 	Client: []
2017-11-23 07:49:57.620  INFO [           main]     i.k.m.config.DnsOverwriteConfig : 	Federation: []
2017-11-23 07:49:57.633  INFO [           main]  i.k.m.invitation.InvitationManager : Loading saved invites
2017-11-23 07:49:57.659  INFO [           main]  i.k.m.invitation.InvitationManager : Setting up invitation mapping refresh timer
2017-11-23 07:49:57.812  INFO [           main]  i.kamax.mxisd.config.SessionConfig : --- Session config ---
2017-11-23 07:49:57.817  INFO [           main]  i.kamax.mxisd.config.SessionConfig : Global Policy: {"validation":{"enabled":true,"forLocal":{"enabled":true,"toLocal":true,"toRemote":{"enabled":true,"server":"root"}},"forRemote":{"enabled":true,"toLocal":false,"toRemote":{"enabled":true,"server":"root"}}}}
2017-11-23 07:49:57.849  INFO [           main]    io.kamax.mxisd.config.ViewConfig : --- View config ---
2017-11-23 07:49:57.857  INFO [           main]    io.kamax.mxisd.config.ViewConfig : Session: {"local":{"onTokenSubmit":{"failure":"session/local/tokenSubmitFailure","success":"session/local/tokenSubmitSuccess"}},"localRemote":{"onTokenSubmit":{"failure":"session/local/tokenSubmitFailure","success":"session/localRemote/tokenSubmitSuccess"}},"remote":{"onRequest":{"failure":"session/remote/requestFailure","success":"session/remote/requestSuccess"},"onCheck":{"failure":"session/remote/checkFailure","success":"session/remote/checkSuccess"}}}
2017-11-23 07:49:57.956  INFO [           main]    i.k.m.directory.DirectoryManager : Directory providers:
2017-11-23 07:50:00.193  INFO [           main] k.m.MatrixIdentityServerApplication : Started MatrixIdentityServerApplication in 11.439 seconds (JVM running for 12.359)
2017-11-23 07:50:40.363  INFO [nio-8090-exec-1]  i.k.m.c.i.v1.SessionRestController : Request POST: http://localhost:8090/_matrix/identity/api/v1/validate/email/requestToken
2017-11-23 07:50:40.384  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : Server 127.0.0.1 is asking to create session for email:[email protected] (Attempt #1) - Next link: http://localhost:8080/#/register?client_secret=B7duTlxeyuuxZSVdzdYUAO4fs5fOmMNO&hs_url=https%3A%2F%2Fidigid.com&is_url=https%3A%2F%2Fidigid.com&session_id=tOymxqxZagAtlbhwGMtyQEOy
2017-11-23 07:50:40.396  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : No existing session for email:[email protected]
2017-11-23 07:50:40.397  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : Is 3PID bound to local domain? false
2017-11-23 07:50:40.401  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : Generated new session 1511423440397 to validate email:[email protected] from server 127.0.0.1
2017-11-23 07:50:40.401  INFO [nio-8090-exec-1]   i.k.mxisd.session.SessionMananger : Session 1511423440397 for email:[email protected]: sending remote-only validation notification
2017-11-23 07:50:40.421 ERROR [nio-8090-exec-1]     i.k.m.c.DefaultExceptionHandler : Unknown error when handling http://localhost:8090/_matrix/identity/api/v1/validate/email/requestToken

java.lang.NullPointerException: null
	at io.kamax.mxisd.threepid.notification.PlaceholderNotificationGenerator.populateForCommon(PlaceholderNotificationGenerator.java:46) ~[classes!/:na]
	at io.kamax.mxisd.threepid.notification.PlaceholderNotificationGenerator.populateForValidation(PlaceholderNotificationGenerator.java:79) ~[classes!/:na]
	at io.kamax.mxisd.threepid.notification.PlaceholderNotificationGenerator.populateForRemoteValidation(PlaceholderNotificationGenerator.java:86) ~[classes!/:na]
	at io.kamax.mxisd.threepid.connector.email.EmailSendGridNotificationHandler.sendForRemoteValidation(EmailSendGridNotificationHandler.java:115) ~[classes!/:na]
	at io.kamax.mxisd.notification.NotificationManager.sendforRemoteValidation(NotificationManager.java:82) ~[classes!/:na]
	at io.kamax.mxisd.session.SessionMananger.create(SessionMananger.java:168) ~[classes!/:na]
	at io.kamax.mxisd.controller.identity.v1.SessionRestController.init(SessionRestController.java:97) ~[classes!/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) [spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

2017-11-23 07:50:40.424  INFO [nio-8090-exec-1]     i.k.m.c.DefaultExceptionHandler : Request POST http://localhost:8090/_matrix/identity/api/v1/validate/email/requestToken - Error M_UNKNOWN: An internal server error occurred. If this error persists, please contact support with reference #1511423440423

requestToken was requested but not implemented

Hi. My plan is to use a single non-federated Matrix-Synapse Server with ID Server for internal using.
The LDAP Lookup for Mail works great. But when a user try to insert a Mailadress in his profile in Riot, there is an Error message on the Server:
http://127.0.0.1:8090/_matrix/identity/api/v1/validate/email/requestToken was requested but not implemented

And in Riot:
{"timestamp":1499097348081,"status":501,"error":"Not Implemented","exception":"io.kamax.mxisd.exception.NotImplementedException","message":"No message available","path":"/_matrix/identity/api/v1/validate/email/requestToken"} (Status 501)

Am i doing something wrong? Sorry for my noobish question.

What i am expecting: The Server is sending an email to the user, to verify the address.

[FEATURE] Add support for Auth via PAM

Just as a feature request.

synapse integration and HTTPS

Run ./gradlew bootRun, then meet some errors and blocked, the log as below:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.4.3.RELEASE)

2017-02-09 11:04:35.853  INFO 11624 --- [           main] i.k.m.MatrixIdentityServerApplication    : Starting MatrixIdentityServerApplication on localhost with PID 11624 (/home/wangjh/work/mxisd/build/classes/main started by wangjh in /home/wangjh/work/mxisd)
2017-02-09 11:04:35.883  INFO 11624 --- [           main] i.k.m.MatrixIdentityServerApplication    : No active profile set, falling back to default profiles: default
2017-02-09 11:04:36.126  INFO 11624 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@13d73f29: startup date [Thu Feb 09 11:04:36 CST 2017]; root of context hierarchy
2017-02-09 11:04:39.902  INFO 11624 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8090 (http)
2017-02-09 11:04:39.932  INFO 11624 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
2017-02-09 11:04:39.936  INFO 11624 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.6
2017-02-09 11:04:40.196  INFO 11624 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2017-02-09 11:04:40.199  INFO 11624 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 4084 ms
2017-02-09 11:04:40.554  INFO 11624 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean  : Mapping servlet: 'dispatcherServlet' to [/]
2017-02-09 11:04:40.561  INFO 11624 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'characterEncodingFilter' to: [/*]
2017-02-09 11:04:40.563  INFO 11624 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2017-02-09 11:04:40.564  INFO 11624 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2017-02-09 11:04:40.565  INFO 11624 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'requestContextFilter' to: [/*]
2017-02-09 11:04:41.929  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@13d73f29: startup date [Thu Feb 09 11:04:36 CST 2017]; root of context hierarchy
2017-02-09 11:04:42.118  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/_matrix/identity/api/v1/store-invite],methods=[POST]}" onto public java.lang.String io.kamax.mxisd.controller.v1.InvitationController.store()
2017-02-09 11:04:42.129  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/_matrix/identity/api/v1/pubkey/ephemeral/isvalid],methods=[GET]}" onto public java.lang.String io.kamax.mxisd.controller.v1.KeyController.checkEphemeralKeyValidity()
2017-02-09 11:04:42.131  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/_matrix/identity/api/v1/pubkey/isvalid],methods=[GET]}" onto public java.lang.String io.kamax.mxisd.controller.v1.KeyController.checkKeyValidity()
2017-02-09 11:04:42.133  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/_matrix/identity/api/v1/pubkey/{keyType}:{keyId}],methods=[GET]}" onto public java.lang.String io.kamax.mxisd.controller.v1.KeyController.getKey(java.lang.String,int)
2017-02-09 11:04:42.137  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/_matrix/identity/api/v1/lookup],methods=[GET]}" onto public java.lang.String io.kamax.mxisd.controller.v1.MappingController.lookup(java.lang.String,java.lang.String)
2017-02-09 11:04:42.142  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/_matrix/identity/api/v1/validate/email/submitToken],methods=[GET || POST]}" onto public java.lang.String io.kamax.mxisd.controller.v1.SessionController.validate()
2017-02-09 11:04:42.144  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/_matrix/identity/api/v1/3pid/bind],methods=[POST]}" onto public java.lang.String io.kamax.mxisd.controller.v1.SessionController.bind()
2017-02-09 11:04:42.146  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/_matrix/identity/api/v1/validate/email/requestToken],methods=[POST]}" onto public java.lang.String io.kamax.mxisd.controller.v1.SessionController.init()
2017-02-09 11:04:42.150  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/_matrix/identity/api/v1/3pid/getValidated3pid],methods=[POST]}" onto public java.lang.String io.kamax.mxisd.controller.v1.SessionController.check()
2017-02-09 11:04:42.163  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2017-02-09 11:04:42.169  INFO 11624 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2017-02-09 11:04:42.263  INFO 11624 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-02-09 11:04:42.265  INFO 11624 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-02-09 11:04:42.367  INFO 11624 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-02-09 11:04:43.063  INFO 11624 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2017-02-09 11:04:43.181  INFO 11624 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8090 (http)
2017-02-09 11:04:43.194  INFO 11624 --- [           main] i.k.m.MatrixIdentityServerApplication    : Started MatrixIdentityServerApplication in 8.619 seconds (JVM running for 10.035)

Allow to configure SQL table prefix for the Wordpress identity store

Currently, wp_ is hardcoded but Wordpress gives the option to set another value during install.
The only option currently is to provide full custom queries.

We should offer the possibility to easily set a different prefix in the configuration.

New Identity store: executable

Create a new Identity store which is backed by executable commands on the system.
The data will be provided either as env or arguments, whichever is set.

Configuration example:

exec:
  auth:
    command: "/path/to/exec"
    env:
      - "USER=$user"
      - "PASS=$pass
    args: [ "--auth", "-u", "$user", "-p", "$pass" ]
    return_code: 
      success: [ 0, 128 ]

To figure out: what output format(s) should we accept

Relates to: #36

test discovery fail

Hi, I tried inviting [email protected] to a room using my own server, aliased here as https://testing.com and failed. Inviting @mxisd-lookup-test:kamax.io works fine, so my Synapse is working fine.

Partial output from sudo systemctl status mxisd:

k.m.l.p.RemoteIdentityServerFetcher : Looking up email 3PID [email protected] using https://kamax.io:443
k.m.l.p.RemoteIdentityServerFetcher : Found 3PID mapping: {"address":"[email protected]","medium":"email","mxid":"@mxisd-lookup-test:kamax.io","not_after":25340230079
i.k.m.invitation.InvitationManager : Mapping for email:[email protected] already exists, refusing to store invite
i.k.m.c.DefaultExceptionHandler : Request POST http://127.0.0.1:8090/_matrix/identity/api/v1/store-invite - Error M_ALREADY_EXISTS: A mapping already exists for this 3PID
i.k.m.c.i.v1.MappingController : Got single lookup request from [my ip] with client Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 
l.s.RecursivePriorityLookupStrategy : Host [my ip] allowed for recursion: false
i.k.m.c.i.v1.MappingController : No mapping was found, return empty JSON object

IPv6 connections break recursive lookups

2017-03-02 16:38:28.866 ERROR 20163 --- [nio-8090-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet]  in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: Could not parse [0:0:0:0: 0:0:0:1]] with root cause
 
java.lang.IllegalArgumentException: Could not parse [0:0:0:0:0:0:0:1]
        at org.apache.commons.net.util.SubnetUtils.toInteger(SubnetUtils.java:287) ~[commons-net-3.5.jar:3.5]
        at org.apache.commons.net.util.SubnetUtils.access$400(SubnetUtils.java:27) ~[commons-net-3.5.jar:3.5]
        at org.apache.commons.net.util.SubnetUtils$SubnetInfo.isInRange(SubnetUtils.java:125) ~[commons-net-3.5.jar:3.5]
        at org.apache.commons.net.util.SubnetUtils$SubnetInfo$isInRange.call(Unknown Source) ~[na:na]
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) ~[groovy-all-2.4.7.jar:2.4.7]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) ~[groovy-all-2.4.7.jar:2.4.7]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) ~[groovy-all-2.4.7.jar:2.4.7]
        at io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy.find(RecursivePriorityLookupStrategy.groovy:76) ~[main/:na]
        at io.kamax.mxisd.lookup.strategy.LookupStrategy$find.call(Unknown Source) ~[na:na]
...

SubnetUtils, which is part of Apache Commons Net, does not support IPv6.

Handle bulk lookup end point

/bulk_lookup is not implemented due to an incomplete IS spec.

Wait for it to be documented or reverse engineer sydent.

Build failure

Hello,

I tried building mxisd but without success on Ubuntu 14.04. Here is the output:

:compileGroovy (Thread[Daemon worker Thread 2,5,main]) started.
:compileGroovy
Putting task artifact state for task ':compileGroovy' into context took 0.0 secs.
Task :compileGroovy class loader hash: 01d9458a7e5b59caccdb01424c57a842
Task :compileGroovy actions class loader hash: 18198991b357e4282b88db10fdb1e992
Executing task ':compileGroovy' (up-to-date check took 0.008 secs) due to:
  No history is available.
Starting process 'Gradle Worker Daemon 1'. Working directory: /home/mxisd/mxisd Command: /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -cp /root/.gradle/caches/3.3/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 1'
Successfully started process 'Gradle Worker Daemon 1'
Started Gradle worker daemon (0.372 secs) with fork options DaemonForkOptions{minHeapSize=null, maxHeapSize=null, jvmArgs=[], classpath=[/root/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy-all/2.4.7/c5371aaa20bcdca1175d9477fc0811f4fd99b68a/groovy-all-2.4.7.jar, /root/.gradle/wrapper/dists/gradle-3.3-bin/64bhckfm0iuu9gap9hg3r7ev2/gradle-3.3/lib/ant-1.9.6.jar, /root/.gradle/wrapper/dists/gradle-3.3-bin/64bhckfm0iuu9gap9hg3r7ev2/gradle-3.3/lib/ant-launcher-1.9.6.jar]}.
Initialized native services in: /root/.gradle/native
Executing org.gradle.api.internal.tasks.compile.ApiGroovyCompiler in worker daemon.
Compiling with JDK Java compiler API.
/home/mxisd/mxisd/src/main/groovy/io/kamax/mxisd/lookup/provider/BridgeFetcher.java:35: error: cannot find symbol
import java.util.Optional;
                ^
  symbol:   class Optional
  location: package java.util
/home/mxisd/mxisd/src/main/groovy/io/kamax/mxisd/lookup/provider/BridgeFetcher.java:49: error: cannot find symbol
    public Optional<?> find(SingleLookupRequest request) {
           ^
  symbol:   class Optional
  location: class BridgeFetcher
/home/mxisd/mxisd/build/tmp/compileGroovy/groovy-java-stubs/io/kamax/mxisd/lookup/fetcher/IBridgeFetcher.java:13: error: cannot find symbol
 Optional<?> find(io.kamax.mxisd.lookup.SingleLookupRequest request);
 ^
  symbol:   class Optional
  location: interface IBridgeFetcher
/home/mxisd/mxisd/build/tmp/compileGroovy/groovy-java-stubs/io/kamax/mxisd/lookup/provider/RemoteIdentityServerFetcher.java:22: error: cannot find symbol
@java.lang.Override() public  Optional<?> find(java.lang.String remote, java.lang.String type, java.lang.String threePid) { return (Optional<?>)null;}
                              ^
  symbol:   class Optional
  location: class RemoteIdentityServerFetcher
/home/mxisd/mxisd/build/tmp/compileGroovy/groovy-java-stubs/io/kamax/mxisd/lookup/fetcher/IRemoteIdentityServerFetcher.java:14: error: cannot find symbol
 Optional<?> find(java.lang.String remote, java.lang.String type, java.lang.String threePid);
 ^
  symbol:   class Optional
  location: interface IRemoteIdentityServerFetcher
/home/mxisd/mxisd/src/main/groovy/io/kamax/mxisd/lookup/provider/BridgeFetcher.java:50: error: cannot find symbol
        Optional<String> mediumUrl = Optional.ofNullable(cfg.getMappings().get(request.getType()));
        ^
  symbol:   class Optional
  location: class BridgeFetcher
/home/mxisd/mxisd/src/main/groovy/io/kamax/mxisd/lookup/provider/BridgeFetcher.java:50: error: cannot find symbol
        Optional<String> mediumUrl = Optional.ofNullable(cfg.getMappings().get(request.getType()));
                                     ^
  symbol:   variable Optional
  location: class BridgeFetcher
/home/mxisd/mxisd/src/main/groovy/io/kamax/mxisd/lookup/provider/BridgeFetcher.java:62: error: cannot find symbol
            return Optional.empty();
                   ^
  symbol:   variable Optional
  location: class BridgeFetcher
/home/mxisd/mxisd/build/tmp/compileGroovy/groovy-java-stubs/io/kamax/mxisd/lookup/provider/RemoteIdentityServerFetcher.java:22: error: cannot find symbol
@java.lang.Override() public  Optional<?> find(java.lang.String remote, java.lang.String type, java.lang.String threePid) { return (Optional<?>)null;}
                                                                                                                                    ^
  symbol:   class Optional
  location: class RemoteIdentityServerFetcher
9 errors
startup failed:
Compilation failed; see the compiler error output for details.

1 error

Exception executing org.gradle.api.internal.tasks.compile.ApiGroovyCompiler in worker daemon: org.gradle.api.internal.tasks.compile.CompilationFailedException: Compilation failed; see the compiler error output for details..
:compileGroovy FAILED
:compileGroovy (Thread[Daemon worker Thread 2,5,main]) completed. Took 3.175 secs.

FAILURE: Build failed with an exception.