aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* util: Do not store "" as the message CHHEADmasterJason Gunthorpe2021-06-241-5/+4
| | | | | | | This was causing assertions if this condition was ever hit. Instead just delete the corrupted message id. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* util: Use a different algorithm for asyncio_complete()Jason Gunthorpe2020-09-251-6/+5
| | | | | | | | The old version only worked on python > 3.7, this should work universally. Allow all the work in the gather list to finish naturally and just propogate any exceptions that might have been generated. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* Do not immediately restore deleted itemsJason Gunthorpe2020-09-251-5/+7
| | | | | | | | | When not using the deleted maildir flag messages would be restored immediately after the cloud was updated. This is because nmsgs was updated in the wrong order. Fixes: 131d182e0054 ("Add a --offline option") Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* Log the non-HTTP error exceptionJason Gunthorpe2020-09-252-4/+4
| | | | | | For debugging. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* O365: Protect the page fetches in get_json_paged() with _retry_protectJason Gunthorpe2020-08-311-2/+6
| | | | | | | Getting a 404 or other error here would explode the whole thing. Instead the fetch of the next URL should be re-issued. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* oauth: Extract the web server address from the setup URLJason Gunthorpe2020-07-281-1/+4
| | | | | | This lets the OAUTH module be used for other localhost ports too. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* cms-oauth: Check the token lifetime before returning itJason Gunthorpe2020-07-172-2/+6
| | | | | | | If the mail functions are not being used then the ticket will get stale, check the timestamp before returning it and refresh if too old. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* doc: Update example-exim4.confJason Gunthorpe2020-07-171-14/+30
| | | | | | | | This simplfies the config to use address_data and adds an extra stanza to handle <> envelope from addresses, generated by cron or internally by exim. This works OK on gmail now. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* gmail: Fix a spelling error in add_message()Jason Gunthorpe2020-07-101-1/+1
| | | | | | | Not sure when gmail sends a message ID without a labelId stanza, but apparently it happens rarely. Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
* OAUTH: Make some sense of the scopesJason Gunthorpe2020-06-223-53/+95
| | | | | | | | With the ability to run as a broker for IMAP/SMTP we can limit the scopes requested based on the configuration. Add a fake _CMS_ protocol that refers to the scopes required to operate internally. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* OAUTH: Add support to get an IMAP OAUTH tokenJason Gunthorpe2020-06-228-31/+130
| | | | | | | Latest mutt can do this for MS and GMail providers, provide support for getting the right scope and some examples how to set it up. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* GMail: Instead of polling each Label, poll the profileJason Gunthorpe2020-06-221-9/+14
| | | | | | | | | The global history_id covers all labels, if we see it change then do a delta query to all the labels to figure things out. This reduces the amount of server traffic if there are multiple labels. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Add debugging to try to find an assertionJason Gunthorpe2020-06-223-0/+11
| | | | | | | | | | | File "/home/jgg/oss/sync/cloud_mdir_sync/maildir.py", line 210, in force_content self._store_msg(msgs[content_hash]) File "/home/jgg/oss/sync/cloud_mdir_sync/maildir.py", line 145, in _store_msg self.msgdb.write_content(cloudmsg.content_hash, msg.fn) File "/home/jgg/oss/sync/cloud_mdir_sync/messages.py", line 279, in write_content assert content_hash in self.file_hashes Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Gmail: Retry on 502 Bad GatewayJason Gunthorpe2020-06-221-0/+1
| | | | | | gmail occasionally returns this. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* OAUTH: Only start one webbrowser tab for concurrent authenticationsJason Gunthorpe2020-06-223-10/+14
| | | | | | | | Starting a browser tab per authentication attempt makes a mess, so long as there is something in auth_redirs then the active browser tab will eventually get redirected to the new auth being added. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* O365: Stop using MSAL for OAUTHJason Gunthorpe2020-06-223-61/+79
| | | | | | | | | | | Since gmail figured out how to use oauth using asyncio and oauthlib, just use it for the O365 flow too. This greatly speeds up refreshing tickets since both graph and OWA scopes can run in parallel. This also makes the dependency list small enough the tool will run with built-in python modules for most distros. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* GMail: Stop using requests-oauth and just invoke the POSTs directlyJason Gunthorpe2020-06-223-88/+158
| | | | | | | aio-http is much better, requests seems to get stuck occasionaly. This allows all GMail accounts to acquire access tokens concurrently. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* O365: Use batching during modificationJason Gunthorpe2020-06-221-23/+97
| | | | | | | | | | | | | | | | | | For whatever reason this week the concurrency limit in Graph went way down for modify, this event impacts the batches. It turns out graph is crazy. You can put up to 20 requests in a batch, but during execution they can fail with throttling and need retry. Experiments suggest 3 is the right number to avoid throttling on modify, for some reason. Documentation seems wrong, and probably something broke recently. Implement batching, run the batch break up sequentially, and limit the batches to 3. Use batching only for modify operations. GET can still use a concurrency limit of 5. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* O365: Log throttling in a more informative wayJason Gunthorpe2020-06-141-1/+7
| | | | | | | | | | For some reason the graph server has started to throttle message modification after 3-5 requests, I assume it is a server bug, so make the logging clearer for now. Also use the response headers to guide the retry timer. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Add cms-oauth wrapperJason Gunthorpe2020-06-141-0/+9
| | | | | | So it can run without installation. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Keep the inode cache in sync when writing to the MailDirJason Gunthorpe2020-05-302-0/+8
| | | | | | | | MailDir writes cause the inode mtime and ctime to change which causes the cache to become invalid. Update the mtime and ctime information after changing it. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Speed up loading large databases on startupJason Gunthorpe2020-05-301-14/+21
| | | | | | Batching the sha1sum work is dramatically faster. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Increase the page size for listing O365 mailboxesJason Gunthorpe2020-05-301-0/+1
| | | | | | | This helps large mailboxes quite a lot, it looks like this can list some 700 messages/second Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Manage concurrency to avoid running out of file descriptorsJason Gunthorpe2020-05-302-29/+33
| | | | | | | | | | | | AIO does not seem to strictly pipeline work, so it is possible for a very large number of message download tasks to be scheduled. Each one that gets scheduled opens a file descriptor, and so we can run out of them fairly fast. Strictly limit the number of open files per mailbox during message downloading. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Add cms-oauthJason Gunthorpe2020-05-285-1/+445
| | | | | | | | | | | This is a command line program to get the OAUTH tokens from the credential server. It is intended to fit into the 'call a program to get the token' methodology that several tools are implementing. Several options are provided to format the token and a built in SMTP protocol tests that the server is working properly. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Add OAUTH Credential serverJason Gunthorpe2020-05-288-15/+349
| | | | | | | | | | | | | | | | The OAUTH credential server allows CMS to ack as an OAUTH broker and supply bearer tokens to other applications in the system. Currently this only support SMTP tokens for outbound mail delivery. A UNIX domain socket is used to communicate between the SMTP agent and CMS. A simple one line protocol is used to specify the account requested and CMS returns the plain XAOUTH2 response string. The agent is responsible to base64 encode it. This works for GMail and O365 mailboxes. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Remove cfg.domainsJason Gunthorpe2020-05-284-55/+42
| | | | | | | | Directly connect the 'account' API objects to their mailbox users through the config language instead of trying to fix it up after the fact with a dictionary. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* o365: Support delete_action in O365Jason Gunthorpe2020-03-271-1/+6
| | | | | | | If set to "delete" the deleted messages will go to Deleted Items by default things go to Archive, just like gmail. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* gmail: Fix determining the start_history_idJason Gunthorpe2020-02-241-7/+13
| | | | | | | | | | | | Occasionally gmail returns history records that have junk in them, for instance messages IDs that don't exist. This appears to be a bug. Using the latest history ID seems to guarantee that a subsequent history query will be empty and we can then avoid the junk and start again. Go back to full query if delta fails for any reason. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Catch failures from the sha1sum forkJason Gunthorpe2020-02-211-1/+2
| | | | | | | | This should trigger a sleep and retry, not a program failure. It happens if the sha1sum races with the user renaming a MailDir file. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Fix an exception if message fetching failsJason Gunthorpe2020-02-212-0/+2
| | | | | | | Since size is used to print the log message, even on failure, set it to some value. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Don't leave asyncio tasks running unexpectedlyJason Gunthorpe2020-02-074-13/+29
| | | | | | | | All cases where gather is called intend that the tasks will complete successfully or all cancel at the first error. Add a little wrapper to achieve this. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Allow access to the message headers during mail directingJason Gunthorpe2020-02-073-30/+90
| | | | | | | Access to the headers like List-ID allows some client side filtering of incoming messages. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Do not pass msgdb all over the placeJason Gunthorpe2020-02-075-39/+38
| | | | | | | The mailboxes can only be linked to a single msgdb, always take it from the mailbox cfg. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Lift Mailbox.cfg out of the sub classesJason Gunthorpe2020-02-076-20/+22
| | | | | | Make sure it is always set Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Update documentationJason Gunthorpe2020-02-071-17/+59
| | | | | | For Gmail and other notes about O365 Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* GMail supportJason Gunthorpe2020-02-066-11/+673
| | | | | | Basic support for GMail using the REST API Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Allow a Mailbox to search for messages with a None email_idJason Gunthorpe2020-02-051-16/+40
| | | | | | | Gmail does not have easy access to the email_id, but does have a stable storage_id, so provide a second dictionary. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Preserve flags in the local MailDir that are not supported in the cloudJason Gunthorpe2020-02-052-7/+24
| | | | | | This is needed for gmail, since it doesn't support all of the flags. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Do not run tracing by defaultJason Gunthorpe2020-02-052-4/+2
| | | | | | trace is only for debugging adventures. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Add some more type annotations to O365MailboxJason Gunthorpe2020-02-042-10/+14
| | | | Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Add a --offline optionJason Gunthorpe2020-01-312-10/+40
| | | | | | | | | | | This allows treating the local message flags as authoritative during program startup. Messages marked locally as deleted will be deleted on the server during startup. For safety, this never considers the absence of a message locally as a deletion. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Document how to setup the Trash modeJason Gunthorpe2020-01-311-0/+13
| | | | Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Support FLAG_DELETED in Office365Jason Gunthorpe2020-01-311-16/+26
| | | | | | | | | | | | | | When a local message is tagged deleted then update the cloud flags and delete the message. Using the Trash flag in a mail dir allows the cloud upload step to see the original message and then upload the local flags before deleting it. For instance, this will allow the Replied flag to be set upon deletion. Once deleted in the cloud the next sync iteration will delete the message locally. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Support the Trash flag in MailDirJason Gunthorpe2020-01-312-3/+8
| | | | | | | This can be set by mutt when 'set maildir_trash=yes' and shows as a deleted item. Keep track of it on the maildir side. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Move Office365 flag manipulation into a functionJason Gunthorpe2020-01-311-67/+70
| | | | | | For reuse in the next patches Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Trigger an update of a O365 mailbox if the notification FD closesJason Gunthorpe2020-01-181-0/+2
| | | | | | Fixes an issue where mailbox changes could be missed. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
* Initial commit of cloud_mdir_syncJason Gunthorpe2020-01-1015-0/+2200
I have been using for a few months now with no ill effects. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-06 15:20:10 +0000