Using since_id and limit not starting at oldest orders?

flexjolyflexjoly Member Posts: 13 

Hi,

Can it be that we miss orders when using since_id and limit together? Or has the behavior for this changed lately?

Example: lets say the lastest imported order has id 12345 and since then there are 10 new orders. If my request is: orders.json?since_id=12345&limit=5, we expected to get the oldest 5 orders directly after 12345...

I implemented this a couple of weeks ago, and I did debug on this! Because I also needed to revert the sort-order of the orders so I could import the oldest as first.

Today our client contacted us, that they missed older orders....😨

When debugging again, it seems indeed that we get the youngest 5 orders, leaving a gap for the 5 older ones. So now I wonder if I did overlook something or not....😳

Is the response for this api-request changed? Or did I miss something?

If this is the normal behavior, can you please give us an option to define the sort-order also? Else we need to make extra (unnecessary) calls to the api to determine which orders to import.


Thanks in advance,

lydia

15 comments

  • gregaricangregarican Member Posts: 333 

    I haven't worked much with the Lightspeed E-Com API, but if it's similar to the Lightspeed Retail API, perhaps there are undocumented query parameters? In Retail there are the order_by and orderby_desc options --> https://developers.lightspeedhq.com/retail/introduction/parameters/. Just a left-field suggestion πŸ˜€

  • gregaricangregarican Member Posts: 333 

    I was able to work something similar, out in my test e-com account, in terms of using orderby and orderby_desc query parameters.


    For example,

    orders.json?since_id=938513&orderby=createdAt&limit=5&orderby_desc=0

  • flexjolyflexjoly Member Posts: 13 

    Thanks for your suggestions!

    I tried it.... but I don't see any difference :-(

    It still gives the youngest orders.

    api.webshopapp.com/nl/orders.json?limit=2&since_id=11111&orderby=createdAt&orderby_desc=1

    or desc=0

    No change :(

  • gregaricangregarican Member Posts: 333 

    orderby_desc=0 would mean don't put the results in descending order. This should provide the oldest orders first.

  • flexjolyflexjoly Member Posts: 13 

    Hi,

    jup! that's why I also wrote 'or desc=0' but that was a bit unclear, sorry.

    I tried both. The order did not differ and both times it still gives the youngest orders. :-(

    greetz

  • Adrian SamuelAdrian Samuel Moderator, Lightspeed Staff Posts: 569 moderator

    Hey @flexjoly,

    By Default when you get orders from eCom you’re getting then in descending order.

    So you might have a list of order IDs such as [5,4,3,2,1] where order 5 is the latest order for the shop.

    When you make the API request: GET orders.json?since_id=2

    The response will be [5,4,3]

    If you add the filter limit = 2

    The response will be [5,4]

    So the since_id gives you the newest orders since that ID. Think of it as saying “since this order, what new orders have I received?”

    The limit filter limits the response to only include the amount of orders you want to limit starting at the newest.

    We unfortunately don’t have a query parameter filters to change the sort order of what is returned by the API.

    I hope this makes things clearer for you!

    Adrian Samuel

    API Integrations Consultant - Strategic Solutions

    Lightspeed HQ

  • flexjolyflexjoly Member Posts: 13 

    Hi Adrian,

    Thanks for you explanation! So I missed something when debugging the first time 😳

    But to me this is not very logical, sorry. When you query something to get any amount of items, after some id, you expect to get the items directly following that id. Else why is it called 'since_id' ?

    If after a busy weekend or holiday, our customer got 300 orders and we would import at the max=250 orders with since_id.... it will leave a gap of 50, which are impossible to get this way.... πŸ˜–

    ..... Feature-request.... please?!?

    Ok, it is not a problem for me to make a workaround for all this. Like first get all order-ids since the last import, and then loading those. Or doing some magic with timestamps. Or a more-frequent cron, or webhook. But it should not be necessary πŸ™„πŸ˜¬

    Any one else having this problem? How did you manage?

    Greetz, lydia

  • flexjolyflexjoly Member Posts: 13 

    uhm... thinking again on this....

    According to your example:

    When you make the API request: GET orders.json?since_id=2&limit=2 The response will be [5,4]

    When 'since_id=2' is not used... the result is the same.... So it does not do much... 😣

    It just seems another limiter besides 'limit'. Like: GET orders.json?since_id=2&limit=10 It would only prefend to give ids 1 and 2 again. But only the new orders after id=2

  • Adrian SamuelAdrian Samuel Moderator, Lightspeed Staff Posts: 569 moderator

    Hey @flexjoly,

    There might be some confusion here but what's exactly what since_id is doing

    You have a list [5,4,3,2,1] where 5 is the latest order and 1 is the first order.

    GET orders.json?since_id=2 will give you [5,4,3] because 3,4,5 are the orders that happened in order since order_id 2.

    Regarding the limit filter, that is correct, it's limiting the response starting with the newest.


    Regarding you desire to get 300 orders.

    I assume you would know how many orders come in a given period of time because you're using timestamps (unless you're basing it purely off ids)

    You would have 1 page of 250 results and another page of 50 (page 2)

    So you could simply page to the second set of results and work backwards.

    If you have 50 orders in that response, the order at the top of the list (e.g. 5) will be the latest order, and the order at the bottom (i.e. 1) will be the oldest in the response.

    All you would need to do is store the first order from there and then do since_id of the oldest order from the last page.

    But if you're working with timestamps you could just page to get all the data and not be concerned with the since_id filter since you're going to have to make 2 API requests to get it all anyhow

    Adrian Samuel

    API Integrations Consultant - Strategic Solutions

    Lightspeed HQ

  • flexjolyflexjoly Member Posts: 13 
    edited August 21

    Hi Adrian,

    Thanks for your reply.

    The page-param seems a nice workaround (😜). But I will not save orders in the wrong sortorder to the database!

    It is not secure! What happens if the process crashes in the middle? We are left with some of the youngest orders saved, missing the other ones. So then you need to check them every time. And we also have to write a procedure to figure out where the gap with the missing orders is.... that is even more difficult! And I hate creating workaround, while it is possible to program it correctly the first time.

    So you always need a list with the oldest order first! Only that will guarantee, that you won't miss an order in between.

    Do you understand my point of view? You explained nicely how the api works, and I dó understand that.

    Can you please consider what I am asking/telling that if since_id worked different, it would simplify things a lot and it would give better guarantees.

    Can you guys please add this to the api?? It cannot be that difficult! Just a param which would query the orderlist in another way.

    It is not because, I cannot program around this. I can! I am a divaeloperπŸ™‹β€β™€οΈ πŸ’»οΈ for about 30years now (started with my cobol programmers degree) πŸ˜ŽπŸ€“πŸ˜

    Pleasssss!

    Greetz, lydia

  • gregaricangregarican Member Posts: 333 

    Couldn't you pull orders based on a timestamp value and then apply the sort order query parameter? I do this in the Retail API and it works well pulling records. Let's say you have a polling process that runs at scheduled times. Use those scheduled times against the timestamp value.

    Regardless the sortby_desc query parameter should work consistently, but doesn't appear to in your case...

  • flexjolyflexjoly Member Posts: 13 
    edited August 21

    Hi Gregarican,

    Thank you so much, for thinking with me! xxx

    I can do that indeed. Only the eCom-api does not have params to set any sortorder in the query. https://developers.lightspeedhq.com/ecom/endpoints/order/

    I think I have to do some workaround with "created_at_[min|max]" 😣And check how many orders there are in that specific period.

    Regardless the sortby_desc query parameter should work consistently... agreed....

  • Adrian SamuelAdrian Samuel Moderator, Lightspeed Staff Posts: 569 moderator

    Hey @flexjoly,

    Thank you for sharing your expertise and insight with on this topic; we do understand the frustration you're experiencing. I love that you're passionate and your arguments are sound.

    As such I've written a feature request to our product team specifying not only your arguments but also how it would be better for developers as a whole.

    Regarding workarounds:

    -When you get a page of 250 orders, do you insert it straight away, would it be possible to get all orders in that day frame and just sort by ID yourself?

    -If you're concerned about your application crashing, you might consider first, getting the count of those orders in that date frame and then pushing each order into a pub-sub queue that sits on a different server as the rest of your app that saves the orders and then outputs the order in the way you desire them. Since you have the count, you could create some logic that you only start doing DB transactions when you've retrieved all the orders and have some retry logic for when you don't.


    I'm sorry we're not able to provide you with the best experience that you desire now, but we hope to give you what you request and more as we grow and learn from our developer base

    Adrian Samuel

    API Integrations Consultant - Strategic Solutions

    Lightspeed HQ

  • flexjolyflexjoly Member Posts: 13 

    Hi Adrian,

    Sorry for not responding anymore.

    Thank you so much for your kind answer 😊 and for writing a feature request.

    Yes I do reverse sort the orders, but in ranges of 30.

    ...pub-sub queue .... good idea! tx.

    request.

    Greetz!

  • flexjolyflexjoly Member Posts: 13 

    Hi Adrian,

    Have you had any response on the feature-request yet?

    Greetz!

Sign In or Register to comment.