Who know me from the past, on my professional side knows that I used to be a Windows guy. Working mainly with Microsoft Exchange and Active Directory as administrator, designer, time-to-time programmer.
Those days are passed. Today I mostly working with Linux as a DevOps in cloud. I changed the Exchange Management Shell, Transport Agents, Management Consoles to CI/CD tools, IaC, Azure, AWS, git repos, artifactories, Kubernetes, and many more.
Most of the times I'm working for a single entity as employee or contractor. On the side of this eventually I have customers of my own.
Back in 2019 I built a solution for one of my own customers, to download e-mails from Office 365 to a folder via POP3. I used Linux, getmail (using POP3) and samba for it. Actually it was a second incarnation of a previous project doing the same in Microsoft Exchange, with a transport agent (the transport agent concept was removed in Office 365).
Asked me to resolve it. This means, the authentication need to be switched from basic to oauth2
Here is the finally working solution. I had some issues needed to resolve during the journey. I will manly left out the detours, to keep the description clear. I done this on an Ubuntu 18.04LTS (the upgrade wasn't on the list this time)
1. The software
The original getmail exists as a package in the Ubuntu, doesn't support oauth2, so it need to be replaced:
Remove the current getmail:
sudo apt-get purge getmail
The getmail6 (
https://getmail6.org/) what is a fork of the original getmail and support oauth2 is a python3 package. Even the python3 is part of the base installation of the Ubuntu, the pip is not:
sudo apt-get install python3-pip
With the pip3 we can install the getmail6:
sudo pip3 install getmail6
2. Azure setup
Login to the azure console (
https://portal.azure.com/) - don't be afraid if you never used it. It will let you in with your Office 365 account.
Select Manage Azure Active Directory (use the search if needed)
Select App Registrations
Select New Registration
Enter a project name, eg "getmail".
Select Accounts in this organizational directory only (Single tenant) from Supported account types
For Redirect URI type, select Web - missing from the getmail6 documentation
Enter http://localhost for Redirect URI
Select Register
From the Overview page take a note of the
Application (client) ID and the
Directory (tenant) ID.
Next create a secret by selecting Certificates and secrets
Select New client secret
Description: password
Expires: select preferred expiry date
Make a note of:
client_secret
It is available by selecting Value - Be aware! You can see this only once. You can't get the value after you navigate off this page ever.
Next add permissions:
Select API permissions, then Add a permission.
Select Microsoft Graph
Select Delegated permissions
Select the following permissions:
- IMAP.AccessAsUser.All
- POP.AccessAsUser.All
- SMTP.Send
Select Add permissions
3. Create authentication json
We need an autentication json file, will be used by the getmail. We need this on the linux machine, to be present. This also need to be writable by the user running the getmail as the content also managed (and frequently canhged) by the getmail. I called it o365,json, but it is up to you:
{"scope": "https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/POP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access",
"user": "<Office 365 user (email)>",
"client_id": "<client_id>",
"client_secret": "<secret>",
"token_uri": "https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token",
"auth_uri": "https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/authorize",
"redirect_uri": "http://localhost"}
The <Office 365 user (email)>, <client_id>, <tenant_id>, <secret> need to be replaced with the text you previously noted.
4. Setup Azure AD connection with your json
I'm using an Ubuntu vm without gui (headless), setting up everything from my Windows desktop. This step according to the getmail6 documentation, need a browser running on the same box where the getmail is installed, what doesn't exists.
The problem is that during the authentication process in the browser, call back to the getmail authentication backend, in our case to the http://localhost (specially to the 8083/tcp port). As in our case the machine running the browser, and the machine running the getmail, two different boxes, we need to do some tricks to resolve it.
Making the Ubuntu box available on the network will not easily resolve the issue as the Azure AD app registration only allows unencrypted http for the localhost. Any other cases you would require a TLS certificate on the Linux box, and my best guess, some modification in the getmail's python code also.
I'm using PuTTY terminal emulator accessing the Linux box.
What I done the resolve this problem, that I redirected the 8083 port of the remote Linux to my Windows machine.
Open the PuTTY
In the Category box under the Connection expand SSH
Select Tunnels
Into the Source port write 8083
Into the Destination write: 127.0.0.1:8083
Select Add
In the Category box select the Session
Type the Host Name as usual and select Open.
5. Execute Azure AD connection
After logged into the Linux box, we need to execute the following:
getmail-gmail-xoauth-tokens --init <full path of the json file created>
This give back an URL what need to be copied into a browser. After the Office 365/Azure login we get a Permissions requested screen, what need to be accepted.
This call back to the Linux box and insert the security access tokens to the json file created previously. The security link between the getmail and the Office 365 now established
6. Setup the getmailrc file
We need to create a getmailrc file (a text file with a name of your choice placed to the .getmail folder)
The retriever session need to look like this:
[retriever]
type=SimpleIMAPSSLRetriever
use_xoauth2 = True
server=outlook.office365.com
username=<Office 365 user (email)>
password_command=("getmail-gmail-xoauth-tokens", "<full path of the json file created>")
Note:
At the time of writing the SimplePOP3SSLRetriever throw an error while try to access mail, so we should stick to the SimpleIMAPSSLRetriever.
For the other sections and further configuration, running, scheduling the getmail consult with the documentation, as it is out of the scope of this article.
