Filters for the API
With our product we offer to synchronise variants, products, invoices and more from Lightspeed to Exact Online. To do this we need to keep track of all that has changed since the last time we checked, so we'll synchronise everything.
A good way to do this is by webhooks. There are however a few difficulties with those. One for instance is that some customers have plugins or workflows that update entities multiple times, over a longer time span. When working with webhooks, we receive multiple hooks which we need to process every time. We could check if something has changed and if that something might be important, but that would mean that we need to track / store more data (something we don't want to if we don't have to).
Another great problem occurs when there are problems with our product. Say we have problems for whatever reason, and the synchronisations of a week went wrong (webhooks got lost, processing data went wrong, it can be anything). In cases like this our solution when working with for instance Magento is very easy: we set the "last updated at" date back to the moment just before the problems started. Then, we start pulling again: API requests that ask to give the top 60 orders that were updated since that date, ascending.
With Lightspeed, we cannot do this due to the fact that we cannot filter. I believe we can ask for entities that are updated after a certain date, but the return we get is by created_at. We simply cannot query the API to give us the "oldest, last updated entity since a date".
I've been in contact with your by email and there have been created internal issues to address our request. Some quotes from those mails:
Me explaining our request:
When someone uses our product, we synchronise for instance all orders for a period of time. For every other webshop we do synchronisations on, we query on “updated_at”. This way, we can easily limit the number of orders we sync per “pull”. It goes like this:
1. Last sync date modified = 2018-01-01 00:00:002.
2. Request: give me the top 60 orders which are updated after 2018-01-01 00:00:00,ordered by the field updated_at
3. I get 60 orders, of which the last has an updated_at value of 2018-01-04 12:31:49
4. Next request: give me the top 60 orders which are updated after 2018-01-04 12:31:49, again ordered_by the updated_at field
Because we have no way to sort the orders we retrieve from Lightspeed (I believe the order is always created_at from newest to oldest), we had to implement the following logic:
1. Last sync datecreated = 2018-01-01 00:00:00
Request: give me thecount of orders that are created since that date (say the count is 1000)
Divide that could by our limit, 1000/60 = 16.66, so we know there are 17 pages
Take the 17th page of the request: give me the orders that are created since that date, with a page size of 60, page 17
We then take those results (40 orders on the last page), because we know those are the oldest orders
Next request will give me page 16 (unless there’s a new order created, the count will return 961, and there’s only 1 order on the last page this time)
This is very frustrating, as it’s impossible to sync back orders which were updated say in 2018-12. We need to set the sync date way back, to 2018-01 for instance, to be sure we get all orders that were updated in 2018-12 (as an order of 2018-10-20 for instance could be updated to a status (shipped) which we use to process the order).
And your reply:
Thank you for detailing your issues for me. I've created some feature requests that our product team will explore. I'll put your ticket on pending for now. When there are updates, you'll receive a notification about the status of these requests