Duplicate Orders Served to API Software

tpbola

Hello! I'm wondering if anyone has seen this issue. We have an API that sends our orders to Postmates for delivery. It seems that Lightspeed is sending duplicate orders to the software. Our developers have suggested creating a database to assign unique identifiers and reject the second copy.

I'm wondering if there isn't a simpler answer though Lightspeed. I tried toggling some of the email notifications but it that piece doesn't appear to interact with this problem.

Adrian Samuel

@tpbola do you have examples of duplicate orders in the system, or is it simply that your app is sending the same order twice? The latter sounds like an app configuration, the former is something we can look into

tpbola

My developer is saying that "Lightspeed is just sending the same order details twice, and so our software thinks it needs to submit another order". Initially I thought it had something to do with the order notifications but toggling off the Order Confirmation email didn't effect the problem. Is there a technical term for the confirmed order element of the API that I could point them to? 

Adrian Samuel

@tpbola Lightspeed doesn't send data to an integration unless it's via a web-hook. Not knowing how that web-hook is configured or the specific shopID in question prevents me from giving any specific answers. 

There are a large number number of things your developer could be experiencing, it's best get them to join the forums and explain here themselves.

Jacob_Bridge

Hello, I am the project manager on the project in question.
We built a piece of middleware that listens for the webhook from Lightspeed when a submitted order with the shipping method selected as Postmates is marked as paid. Once received we post the order to Postmates - the problem is that the webhook is sending the order details a 2nd time once the order is marked as complete and our middleware has no way of differentiating between a new order and a duplicate, unless we add a database that stores the unique order id so we know not to send the same order twice. This option will cost a bit more for the client as we have to build that database and the logic, for obvious reasons, he is trying to avoid this.

Adrian Samuel

@Jacob_Bridge thank you for adding that added detail!

The order webhook you've implemented must be listening for all events For orders there are 5 events you can subscribe to. These are:

Created
Updated
Deleted
Paid
Shipped

If you want to only listen for the paid event then I would suggest only subscribing to the "paid" event to avoid getting that same data sent to you from the "updated" event.

tpbola

Hi @Adrian Samuel I've heard back from @Jacob_Bridge engineer. They advised that their software is only listening to the "Paid" event and are still receiving two instances of each order. Is there a way to confirm or deny whether the webhooks are functioning correctly?

DocUnid

@Adrian Samuel @tpbola

We experience something simluar: we use multi-shop and webhooks to trigger additional functionality. The webhook (in this case for updated variant) however seems to trigger multiple times and more-or-less randomly when we expect only one trigger. Webhooks even get triggered during nighttime when it is highly unlikely anyone is updating a variant. See screenshots.

Can you enlighten us on how many times a webhook gets triggered for one action? Any batch runs we don't know off?

tpbola

@Adrian Samuel Are you able to look at this? I'm waiting on moving forward on a work-around until we find out whether this issue is on Lightspeed's side

Adrian Samuel

Hey @tpbola @DocUnid, we're not aware of any present issues of this happening in our API implementation.

It's hard to diagnose the root of these issues without having a more intimate knowledge of how your code engages with our software.

In the next instance you get a duplicate payload via a webhook, could  you both send those payloads along with your shopID in my inbox and I will try my best to track these elements.

DocUnid

np. i've got the examples at the ready. just DM-d them to you

tpbola

Hi @Adrian Samuel , where you able to look at @DocUnid 's data? This is the last piece in our development cycle and I have it on hold pending further information on this issue. Thanks

Adrian Samuel

Hey @tpbola, I've been working on reproducing these scenarios examples and I've not been able to reproduce the duplicate mark as paid event being produced twice.

If this persists, I'd be happy to file a bug but if we can't reproduce, then we can't fix.

Regarding the @DocUnid, I've been able to reproduce duplicate webhooks fires just by hitting the save button in the backoffice for the updated variant event because the timeStamp is updated. In addition, the variant would be updated based upon a sale.

From the time stamps provided it doesn't seem unreasonable for someone to place order/make some meta changes and or just hit the save button at those times to cause that webhook to fire

DocUnid

@Adrian Samuel that reasoning is completely logical but not what we see happening nor trying to convey, namely the updated variant webhook triggers multiple (not just two) times during a very short time period. today I witnessed a variant triggering the webhook 3 times in rapid succession within minute, without any change to the delivered payload.

Adrian Samuel

Hey @DocUnid, @tpbola, @Jacob_Bridge so today I've had some success reproducing this. It looks like webhooks go through a cycle of retries as it posts data. 

I had a situation where I inserted an invalid webhook into my webhook endpoint (was a typo) and naturally I didn't receive any data for the configured event.

However when I fixed my URL and then triggered another event, it sent me the payload from the event and then about a minute later it sent two other payloads from those previous failed attempts of webhook reception.

To use webhooks and be on the safe side I would suggest you persist a version of the time stamp and payload and potentially compare to the previous timestamp you have against the ID of the variant/sale etc.

 Add some business logic which removes the data but keeps the IDs for checking purposes. Something like MongoDB, Redis or Memecache might be useful for this.

DocUnid

hi @Adrian Samuel good to hear, and thx for sticking with the challenge. your reply makes me wonder if our server adheres to requirements set by lightspeed api (The lighstpeed server expects a webhook to respond with a 2xx HTTP-status (anything between 200 and 299) within 5 seconds. If the webhook responds with any other status, the call will be considered as failed. The server will retry a webhook multiple times with increasing delays within the next 24 hours. In total the webhook will be tried a maximum of 10 times.). i will have a peek at our response(time)s. is it safe to assume lightspeed api logs response times, and you perhaps have access to those?        

Adrian Samuel

@DocUnid, no worries! Very interesting! Yes, we keep logs for up to 7 days so if you want to throw some particular eCom sales my way, I can look them up

gregarican

Chiming in here. As a test I went into the LS E-Com web client, created a quote, and converted it to an order. I am subscribing to all product events and all order events. A total of 4 webhook calls were sent back to me. 

Below is my log (UTC timestamps). Account ID is 176625. Hope this helps!

2019-02-06 16:45:50 DCH-WEBAPI POST /help X-ARR-LOG-ID=4fca8bf7-6584-4117-9e2c-e6d6fdce4412 443 - 52.20.153.66 WebshopappApi - - dch-webapi.azurewebsites.net 200 0 0 34537 13913 14751

2019-02-06 16:45:50 DCH-WEBAPI POST /help X-ARR-LOG-ID=11c1a943-457e-4934-82f2-4394445c5133 443 - 52.20.153.66 WebshopappApi - - dch-webapi.azurewebsites.net 200 0 0 34537 13913 14751

2019-02-06 16:46:45 DCH-WEBAPI POST /help X-ARR-LOG-ID=d3cd8cbc-49eb-4b1b-a62b-d870c3acf9c2 443 - 52.20.153.66 WebshopappApi - - dch-webapi.azurewebsites.net 200 0 0 34537 14200 15

2019-02-06 16:46:45 DCH-WEBAPI POST /help X-ARR-LOG-ID=f8e022c7-fd8b-4f7c-99e2-d5f87ea63836 443 - 52.20.153.66 WebshopappApi - - dch-webapi.azurewebsites.net 200 0 0 34537 14200 15

Adrian Samuel

Thank you @gregarican, I was able to test just now and I also received a total of 4 webhooks though I may not have followed the same steps as you did. Did you mark the order as paid?

gregarican

Nope, just converted the quote to an order. I'm wondering if I marked as paid if that too would send back a double triggered call. Will test now!

gregarican

I flagged the order as being paid in the LS E-com web client. A duplicate webhook call was triggered. Here is a snippet from my logs below with the specifics.

2019-02-07 13:29:35 DCH-WEBAPI POST /help X-ARR-LOG-ID=59fea0ca-6381-4135-ad74-2bb1e3518f72 443 - 52.20.153.66 WebshopappApi - - dch-webapi.azurewebsites.net 200 0 0 34537 14467 502

2019-02-07 13:29:35 DCH-WEBAPI POST /help X-ARR-LOG-ID=6dace7e7-dc62-4b3e-a65d-7449ee9bfade 443 - 52.20.79.14 WebshopappApi - - dch-webapi.azurewebsites.net 200 0 0 34537 14468 523