Migrating email from one service to another
I'm currently working on migrating from OVH's pro email service to Gsuite.
Experience with OVH's mail service
My monthly cost with OVH's service has been £2.49 since last year.
I've started using Mail Pro since end of September last year.
From my experience, it has been very reliable, however I've been having email delivery issues to Microsoft and Gmail email addresses.
Also, I couldn't find a way to setup DKIM/DMARC with this service.
However, I've had the following SPF record, but I'm not sure how much it helped with email delivery rates.
v=spf1 a mx include:mx.ovh.com -all
My reasons to migrating to Google Gsuite
Well, email delivery is the biggest one.
I rarely send emails but I got the feeling over the past year that people haven't been receiving/reading them.
Second, I believe I can setup DKIM/DMARC records so this will help with email delivery.
Third, is having access to Gsuite's presumably unlimited Google drive service, even with one user. Who knows how long this will last.
Fourth, I'd like to try something new.
Fifth, it's the experience with Gsuite. I've seen people mentioning on different IT news sites/forums of using Gsuite. By using it personally, I might gain valuable experience in a future role.
The only two downsides I can think of are:
- increased monthly costs
- hosting personal email with Google
Migration experience for a personal/single user domain
First, I had to add a txt record to my DNS zone at Cloudflare to prove ownership of my domain.
After, I've done that, I had to start adding users. The 14 days trial period allows I believe adding 10 users if needed for testing.
Since it was just for me, I've already had my user setup.
I had a gotcha here however.
I've got a different Google account with my primary email address which is for testing Google Cloud's trial year with the $300 credit(~£226 at the time).
Since my email was already in use in Google's system, I had to use another email for the initial account creation with Gsuite.
So instead of my old/regular email of whatever@domain.tld, I had to use whatever1@domain.tld.
Fortunately, in the admin panel/user settings I could setup email aliases, so my old address will be an alias of my new address.
I'm still exploring the admin control panel, so that will some take time getting used to.
Imap on Gmail
I use Thunderbird(TB) 68.x on Debian 10, so I need IMAP access to my email address.
I had to turn on IMAP access for the domain, as per kb105694.
Then enable IMAP access for "less secure devices" in Gmail.
I added a "New account" in Thunderbird, with the Gmail imap and smtp records.
Then TB opened up a new window with a Google Oauth page asking me to grant Thunderbird access to my Gmail account, which I did.
Before migrating emails over, I kept my old email address at OVH in TB.
Made 3 local folders for backup purposes.
One for the main inbox, one for sent and for an archived folder.
Migrating 5000+ emails
I've got over 5000 emails gathered over the last few years.
The first thing I've tried was trying to copy from local folders to Inbox, Sent and Archived folder inside Thunderbird itself.
However, this turned out to be unreliable with Gmail because only a fraction of my emails would be uploaded to Gmail. I'd get "Connection timed out to imap.gmail.com" error messages.
Then I've tried the Data Migration wizard inside the Gsuite admin panel, however this turned out to be way too slow.
I aborted the Migration after around 6%.
Then I remembered Imapsync from Gilles Lamiral.
I also remember that he had a docker image up at gilleslamiral/imapsync.
I've pulled down his image, then ran an instance of it and put into background:
docker run --name im -d -t --rm gilleslamiral/imapsync sleep infinity
Then attached into the container called im inside a tmux session:
docker exec -ti im bash
Once inside the container, I started the imapsync process with:
imapsync \
--noreleasecheck \
--host1 pro1.mail.ovh.net --ssl1 --port1 993 --user1 email@domain.tld --password1 hunter2 \
--gmail2 --user2 email1@domain.tld --password2 hunter3
This resulted in the following:
Host1: Expunging folder archive_folder
ETA: Tue Nov 19 19:00:33 2019 1 s 1/5799 msgs left
++++ End looping on each folder
Folders sizes after the synchronization.
You can remove this foldersizes listing by using "--nofoldersizesatend"
...
Host1 Nb folders: 16 folders
Host2 Nb folders: 14 folders
Host1 Nb messages: 5798 messages
Host2 Nb messages: 5709 messages
Host1 Total size: 289009755 bytes (275.621 MiB)Host2 Total size: 200927369 bytes (191.619 MiB) [0/1889]
Host1 Biggest message: 20405072 bytes (19.460 MiB)
Host2 Biggest message: 27913861 bytes (26.621 MiB)
Time spent on sizing: 33.8 seconds
++++ Statistics
Transfer started on : Tue Nov 19 17:32:36 2019
Transfer ended on : Tue Nov 19 19:01:07 2019
Transfer time : 5311.1 sec
Folders synced : 16/16 synced
Messages transferred : 5705
Messages skipped : 92
Messages found duplicate on host1 : 48
Messages found duplicate on host2 : 0
Messages found crossduplicate on host2 : 0
Messages void (noheader) on host1 : 0
Messages void (noheader) on host2 : 0
Messages found in host1 not in host2 : 1 messages
Messages found in host2 not in host1 : 1 messages
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 200801589 (191.499 MiB)
Total bytes skipped : 13109007 (12.502 MiB)
Message rate : 1.1 messages/s
Average bandwidth rate : 36.9 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption at the end : 275.2 MiB (started with 218.4 MiB)
Load end is : 0.18 0.17 0.21 6/659 on 2 cores
Biggest message : 27913861 bytes (26.621 MiB)
Memory/biggest message ratio : 10.3
Start difference host2 - host1 : -5750 messages, -326607299 bytes (-311.477 MiB)
Final difference host2 - host1 : -89 messages, -88082386 bytes (-84.002 MiB)
The sync is not finished, there are 1 identified messages in host1 that are not on host2.
There is no unidentified message
The sync is not strict, there are 1 messages in host2 that are not on host1. Use --delete2 to delete them and have a strict sync. (5679 identified messages in host2)
Detected 3 errors
Check if a new imapsync release is available by adding --releasecheck
Homepage: https://imapsync.lamiral.info/
++++ Listing 3 errors encountered during the sync ( avoid this listing with --noerrorsdump ).
Err 1/3: Host2 Folder Outbox: Could not select: 31 NO [NONEXISTENT] Unknown Mailbox: Outbox (now in authenticated state) (Failure)
Err 2/3: Host2 Folder archive_folder: Could not select: 38 NO [NONEXISTENT] Unknown Mailbox: archive_folder (now in authenticated state) (Failure)
Err 3/3: - msg INBOX/5329 {23820} could not append ( Subject:[whatever], Date:["15-Oct-2017 12:13:35 +0100"], Size:[34930], Flags:[\Seen] ) t
o folder INBOX: * BYE Invalid character in literal (4x)
Exiting with return value 111 (EXIT_WITH_ERRORS) 3/50 nb_errors/max_errors
Many thanks for Gilles Lamiral for this brilliant tool.
Other links that I found useful for Gsuite/Gmail & Imap
- https://support.google.com/a/answer/9003945?hl=en
- https://support.google.com/a/answer/33327?hl=en
- Setting up SPF
- Setting up DKIM
- Generate the DKIM key and txt DNS record in Gsuite admin console
- Turn on DMARC
SPF txt record:
v=spf1 include:_spf.google.com -all
Example DKIM record for a root domain:
DNS Host name (TXT record name): google._domainkey TXT record value: v=DKIM1; k=rsa; p=whatever
Example DMARC record for a root domain:
TXT record name: _dmarc.domain.tld TXT record value: v=DMARC1; p=none; rua=mailto:dmarc@domain.tld
Imap & SMTP details for Gsuite:
Incoming mail settings Server: imap.gmail.com Require SSL: Yes Port: 993 Outgoing mail Server: smtp.gmail.com Require SSL: Yes Require TLS: Yes Require authentication: Yes Port for SSL: 465 Port for TLS/STARTTLS: 587
After finally mucking around with the SPF, DKIM and DMARC records for hours, I get passes on all records at Gmail, Zoho and Outlook. Yay.
Example Terraform Code for Cloudflare provider ~v1.x
Here's a sample code repo:
Google Drive for Rclone
I plan on using Gdrive with Rclone with a crypted remote for remote backups to the cloud.
My current local backup procedure isn't the best, so adding Gdrive to the mix might help with reorganising my thinking & scripts.
To-dos:
- write backup scripts
- setup rclone remotes
- decide whether to backup from both my desktop and nas or just from the nas