Back to Exchange - sort of...

Who know me from the past, on my professional side knows that I used to be a Windows guy. Working mainly with Microsoft Exchange and Active Directory as administrator, designer, time-to-time programmer.

Those days are passed. Today I mostly working with Linux as a DevOps in cloud. I changed the Exchange Management Shell, Transport Agents, Management Consoles to CI/CD tools, IaC, Azure, AWS, git repos, artifactories, Kubernetes, and many more.

Most of the times I'm working for a single entity as employee or contractor. On the side of this eventually I have customers of my own.

Back in 2019 I built a solution for one of my own customers, to download e-mails from Office 365 to a folder via POP3. I used Linux, getmail (using POP3) and samba for it. Actually it was a second incarnation of a previous project doing the same in Microsoft Exchange, with a transport agent (the transport agent concept was removed in Office 365).

A week ago, my friend who took over the role of the external consultant at this company called me, that even he warned those guys months ago, they didn't take a step. The issue is that Microsoft disable the basic authentication on POP3, IMAP4 and SMTP (https://learn.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/deprecation-of-basic-authentication-exchange-online).

Asked me to resolve it. This means, the authentication need to be switched from basic to oauth2

Here is the finally working solution. I had some issues needed to resolve during the journey. I will manly left out the detours, to keep the description clear. I done this on an Ubuntu 18.04LTS (the upgrade wasn't on the list this time)

1. The software

The original getmail exists as a package in the Ubuntu, doesn't support oauth2, so it need to be replaced:

Remove the current getmail:

sudo apt-get purge getmail

The getmail6 (https://getmail6.org/) what is a fork of the original getmail and support oauth2 is a python3 package. Even the python3 is part of the base installation of the Ubuntu, the pip is not:

sudo apt-get install python3-pip

With the pip3 we can install the getmail6:

sudo pip3 install getmail6

2. Azure setup

We need an Enterprise Application for logging in with oauth2 to the Exchange online (the description here is based on the description in the getmail6 documentation https://github.com/getmail6/getmail6/blob/master/docs/getmailrc-examples).

Login to the azure console (https://portal.azure.com/) - don't be afraid if you never used it. It will let you in with your Office 365 account.

Select Manage Azure Active Directory (use the search if needed)

Select App Registrations

Select New Registration

Enter a project name, eg "getmail".

Select Accounts in this organizational directory only (Single tenant) from Supported account types

For Redirect URI type, select Web - missing from the getmail6 documentation

Enter http://localhost for Redirect URI

Select Register

From the Overview page take a note of the Application (client) ID and the Directory (tenant) ID.

Next create a secret by selecting Certificates and secrets

Select New client secret

Description: password

Expires: select preferred expiry date

Make a note of:

client_secret

It is available by selecting Value - Be aware! You can see this only once. You can't get the value after you navigate off this page ever.

Next add permissions:

Select API permissions, then Add a permission.

Select Microsoft Graph

Select Delegated permissions

Select the following permissions:

- IMAP.AccessAsUser.All

- POP.AccessAsUser.All

- SMTP.Send

Select Add permissions

3. Create authentication json

We need an autentication json file, will be used by the getmail. We need this on the linux machine, to be present. This also need to be writable by the user running the getmail as the content also managed (and frequently canhged) by the getmail. I called it o365,json, but it is up to you:

{"scope": "https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/POP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access",

  "user": "<Office 365 user (email)>",

  "client_id": "<client_id>",

  "client_secret": "<secret>",

  "token_uri": "https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token",

  "auth_uri": "https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/authorize",

  "redirect_uri": "http://localhost"}

The <Office 365 user (email)>, <client_id>, <tenant_id>, <secret> need to be replaced with the text you previously noted.

4. Setup Azure AD connection with your json

I'm using an Ubuntu vm without gui (headless), setting up everything from my Windows desktop. This step according to the getmail6 documentation, need a browser running on the same box where the getmail is installed, what doesn't exists.

The problem is that during the authentication process in the browser, call back to the getmail authentication backend, in our case to the http://localhost (specially to the 8083/tcp port). As in our case the machine running the browser, and the machine running the getmail, two different boxes, we need to do some tricks to resolve it.

Making the Ubuntu box available on the network will not easily resolve the issue as the Azure AD app registration only allows unencrypted http for the localhost. Any other cases you would require a TLS certificate on the Linux box, and my best guess, some modification in the getmail's python code also.

I'm using PuTTY terminal emulator accessing the Linux box.

What I done the resolve this problem, that I redirected the 8083 port of the remote Linux to my Windows machine.

Open the PuTTY

In the Category box under the Connection expand SSH

Select Tunnels

Into the Source port write 8083

Into the Destination write: 127.0.0.1:8083

Select Add

In the Category box select the Session

Type the Host Name as usual and select Open.

5. Execute Azure AD connection

After logged into the Linux box, we need to execute the following:

getmail-gmail-xoauth-tokens --init <full path of the json file created>

This give back an URL what need to be copied into a browser. After the Office 365/Azure login we get a Permissions requested screen, what need to be accepted.

This call back to the Linux box and insert the security access tokens to the json file created previously. The security link between the getmail and the Office 365 now established

6. Setup the getmailrc file

We need to create a getmailrc file (a text file with a name of your choice placed to the .getmail folder)

The retriever session need to look like this:

[retriever]

type=SimpleIMAPSSLRetriever

use_xoauth2 = True

server=outlook.office365.com

username=<Office 365 user (email)>

password_command=("getmail-gmail-xoauth-tokens", "<full path of the json file created>")

Note:

At the time of writing the SimplePOP3SSLRetriever throw an error while try to access mail, so we should stick to the SimpleIMAPSSLRetriever.

For the other sections and further configuration, running, scheduling the getmail consult with the documentation, as it is out of the scope of this article.