aboutsummaryrefslogtreecommitdiffstats
path: root/cloud_mdir_sync/office365.py
Commit message (Collapse)AuthorAgeFilesLines
* Log the non-HTTP error exceptionJason Gunthorpe2020-09-251-2/+2
| | | | | | For debugging. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* O365: Protect the page fetches in get_json_paged() with _retry_protectJason Gunthorpe2020-08-311-2/+6
| | | | | | | Getting a 404 or other error here would explode the whole thing. Instead the fetch of the next URL should be re-issued. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* cms-oauth: Check the token lifetime before returning itJason Gunthorpe2020-07-171-1/+3
| | | | | | | If the mail functions are not being used then the ticket will get stale, check the timestamp before returning it and refresh if too old. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* OAUTH: Make some sense of the scopesJason Gunthorpe2020-06-221-41/+70
| | | | | | | | With the ability to run as a broker for IMAP/SMTP we can limit the scopes requested based on the configuration. Add a fake _CMS_ protocol that refers to the scopes required to operate internally. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* OAUTH: Add support to get an IMAP OAUTH tokenJason Gunthorpe2020-06-221-3/+7
| | | | | | | Latest mutt can do this for MS and GMail providers, provide support for getting the right scope and some examples how to set it up. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Add debugging to try to find an assertionJason Gunthorpe2020-06-221-0/+5
| | | | | | | | | | | File "/home/jgg/oss/sync/cloud_mdir_sync/maildir.py", line 210, in force_content self._store_msg(msgs[content_hash]) File "/home/jgg/oss/sync/cloud_mdir_sync/maildir.py", line 145, in _store_msg self.msgdb.write_content(cloudmsg.content_hash, msg.fn) File "/home/jgg/oss/sync/cloud_mdir_sync/messages.py", line 279, in write_content assert content_hash in self.file_hashes Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* OAUTH: Only start one webbrowser tab for concurrent authenticationsJason Gunthorpe2020-06-221-5/+0
| | | | | | | | Starting a browser tab per authentication attempt makes a mess, so long as there is something in auth_redirs then the active browser tab will eventually get redirected to the new auth being added. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* O365: Stop using MSAL for OAUTHJason Gunthorpe2020-06-221-57/+79
| | | | | | | | | | | Since gmail figured out how to use oauth using asyncio and oauthlib, just use it for the O365 flow too. This greatly speeds up refreshing tickets since both graph and OWA scopes can run in parallel. This also makes the dependency list small enough the tool will run with built-in python modules for most distros. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* O365: Use batching during modificationJason Gunthorpe2020-06-221-23/+97
| | | | | | | | | | | | | | | | | | For whatever reason this week the concurrency limit in Graph went way down for modify, this event impacts the batches. It turns out graph is crazy. You can put up to 20 requests in a batch, but during execution they can fail with throttling and need retry. Experiments suggest 3 is the right number to avoid throttling on modify, for some reason. Documentation seems wrong, and probably something broke recently. Implement batching, run the batch break up sequentially, and limit the batches to 3. Use batching only for modify operations. GET can still use a concurrency limit of 5. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* O365: Log throttling in a more informative wayJason Gunthorpe2020-06-141-1/+7
| | | | | | | | | | For some reason the graph server has started to throttle message modification after 3-5 requests, I assume it is a server bug, so make the logging clearer for now. Also use the response headers to guide the retry timer. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Increase the page size for listing O365 mailboxesJason Gunthorpe2020-05-301-0/+1
| | | | | | | This helps large mailboxes quite a lot, it looks like this can list some 700 messages/second Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Manage concurrency to avoid running out of file descriptorsJason Gunthorpe2020-05-301-13/+15
| | | | | | | | | | | | AIO does not seem to strictly pipeline work, so it is possible for a very large number of message download tasks to be scheduled. Each one that gets scheduled opens a file descriptor, and so we can run out of them fairly fast. Strictly limit the number of open files per mailbox during message downloading. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Add OAUTH Credential serverJason Gunthorpe2020-05-281-6/+23
| | | | | | | | | | | | | | | | The OAUTH credential server allows CMS to ack as an OAUTH broker and supply bearer tokens to other applications in the system. Currently this only support SMTP tokens for outbound mail delivery. A UNIX domain socket is used to communicate between the SMTP agent and CMS. A simple one line protocol is used to specify the account requested and CMS returns the plain XAOUTH2 response string. The agent is responsible to base64 encode it. This works for GMail and O365 mailboxes. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Remove cfg.domainsJason Gunthorpe2020-05-281-30/+20
| | | | | | | | Directly connect the 'account' API objects to their mailbox users through the config language instead of trying to fix it up after the fact with a dictionary. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* o365: Support delete_action in O365Jason Gunthorpe2020-03-271-1/+6
| | | | | | | If set to "delete" the deleted messages will go to Deleted Items by default things go to Archive, just like gmail. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Fix an exception if message fetching failsJason Gunthorpe2020-02-211-0/+1
| | | | | | | Since size is used to print the log message, even on failure, set it to some value. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Don't leave asyncio tasks running unexpectedlyJason Gunthorpe2020-02-071-3/+4
| | | | | | | | All cases where gather is called intend that the tasks will complete successfully or all cancel at the first error. Add a little wrapper to achieve this. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Do not pass msgdb all over the placeJason Gunthorpe2020-02-071-7/+7
| | | | | | | The mailboxes can only be linked to a single msgdb, always take it from the mailbox cfg. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Lift Mailbox.cfg out of the sub classesJason Gunthorpe2020-02-071-4/+4
| | | | | | Make sure it is always set Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* GMail supportJason Gunthorpe2020-02-061-1/+1
| | | | | | Basic support for GMail using the REST API Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Preserve flags in the local MailDir that are not supported in the cloudJason Gunthorpe2020-02-051-0/+4
| | | | | | This is needed for gmail, since it doesn't support all of the flags. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Add some more type annotations to O365MailboxJason Gunthorpe2020-02-041-10/+13
| | | | Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Support FLAG_DELETED in Office365Jason Gunthorpe2020-01-311-16/+26
| | | | | | | | | | | | | | When a local message is tagged deleted then update the cloud flags and delete the message. Using the Trash flag in a mail dir allows the cloud upload step to see the original message and then upload the local flags before deleting it. For instance, this will allow the Replied flag to be set upon deletion. Once deleted in the cloud the next sync iteration will delete the message locally. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Move Office365 flag manipulation into a functionJason Gunthorpe2020-01-311-67/+70
| | | | | | For reuse in the next patches Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Trigger an update of a O365 mailbox if the notification FD closesJason Gunthorpe2020-01-181-0/+2
| | | | | | Fixes an issue where mailbox changes could be missed. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Initial commit of cloud_mdir_syncJason Gunthorpe2020-01-101-0/+637
I have been using for a few months now with no ill effects. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-10 22:00:08 +0000