API Connection Timeout?

crallscralls Member Posts: 12
So I'm connecting to the Retail API using the Item endpoint to pull our catalog. I'm able to pull 100 products at a time but we get to a certain point and the request just times out. Are we hitting a rate limit? I'm not able to tell because there is no response from the API. How many requests per minute can we make? Is there a log somewhere on the lightspeed side that can give me more information about why exactly the API stops responding?

20 comments

  • crallscralls Member Posts: 12
    More info, looking at the API response I'm almost getting the entire catalog. My bucket is always at 59 so I don't think it's a rate limit issue but for some reason when I request products 7500 to 7600 the API times out and doesn't send me a response.

    Array
    (
        [count] => 7854
        [offset] => 7400
        [limit] => 100
        [free_bucket] => 59
    )
  • jamesratcliffejamesratcliffe Administrator, Lightspeed Staff Posts: 160 moderator
    @cralls There's something about one or more of the items in that range that's making them take longer to load.

    You might get the response if you raise your timeout limit. You can also try using a limit of 50.
    James Ratcliffe
    Lightspeed HQ
  • crallscralls Member Posts: 12
    That's what I figured. I need your help figuring out which item is causing the problem. I can't figure it out without some kind of a log or some kind of feedback from that end. All I'm getting is a time out. I've tried raising our timeout to 60 seconds between requests and still have the same issue so this isn't any kind of a rate-limiting issue.
  • gregaricangregarican Member Posts: 708 
    When I run into things like that I keep "cutting in half." Try pulling only records 7500 through 7550. If that works, then pull 7551 through 7600. Then once that fails (assuming it would) then pull 7551 through 7575. If that works then pull 7576 through 7600. And so on. Looking at maybe 5-6 iterations to figure out which record is causing an issue. 

    Clunky, but if you are kind of working against a black box of sorts that's about the only suggestion I can think of.
  • jamesratcliffejamesratcliffe Administrator, Lightspeed Staff Posts: 160 moderator
    @cralls Send me the account ID in a private message; I can look into it.

    In many cases, there isn't actually anything wrong with the item(s), they just have more data attached to them (usually related data, especially custom field values). In these cases, you just have to split that page of 100 into smaller chunks.
    James Ratcliffe
    Lightspeed HQ
  • crallscralls Member Posts: 12
    I try to limit of 50. Then I try to limit of 25. Still no luck. I actually didn't even make it as far as that same range when I did this.
  • gregaricangregarican Member Posts: 708 
    Can you run a Fiddler trace to see what's going on? That's the oddest thing. Do you have a lot of custom fields in the recordset? Also, is your code iterating through the paginated results? If so, perhaps just try curl'ing the specific offset and record count you are looking for. Never run into this on my end...
  • jamesratcliffejamesratcliffe Administrator, Lightspeed Staff Posts: 160 moderator
    I don't think I've seen a case where a call with limit=50 or less failed because of resource limitations.

    Can you send me the account ID in a private message?
    James Ratcliffe
    Lightspeed HQ
  • crallscralls Member Posts: 12
    So it seems I cannot grab ANY products after 6000. I am iterating through 10 at a time at this point and when it gets to 6000 I get "Read timed out after 10 seconds" in my log i see the following:

    [2018-09-06 20:21:54] main.CRITICAL: Read timed out after 10 seconds {"exception":"[object] (Zend_Http_Client_Adapter_Exception(code: 1000): Read timed out after 10 seconds at /home/discountgamesinc/public_html/vendor/magento/zendframework1/library/Zend/Http/Client/Adapter/Socket.php:512)"} []

    If I start at 6000 and try to move forward from there it times out right away so basically it seems the API is limited to dealing with the first 6000 products in your catalog?
  • gregaricangregarican Member Posts: 708 
    Uh oh. Just reading this. I'll definitely keep an eye out on this. We are currently at around 4,300 products. But are looking to convert a few other of our companies. Which should put us closer to around 19-20K products when all is said and done. @Michael Carey looping you in since you are the expert on these matters :smile:
  • jamesratcliffejamesratcliffe Administrator, Lightspeed Staff Posts: 160 moderator
    @cralls I tried calls on your account with offsets greater than 6000 with no issues. The response time is pretty high when loading custom field values, though.

    For example, this call took 13.261 seconds but was successful:
    /API/Account/XXXX/Item.json?load_relations=["CustomFieldValues"]&offset=6100<br>
    The response is faster with a smaller page size. This call took 5.750 seconds:
    /API/Account/XXXX/Item.json?load_relations=["CustomFieldValues"]&offset=6100&limit=50<br>
    It looks like your more recent items just have more custom field values set, so the calls are more intensive.


    There are no limits in place related to the total number of items. In practice, you might see performance suffer in the frontend if you have a lot of items, but only as you start to approach 100k. 50k or less will be fine.


    So far this doesn't look like an issue on our end, but I'll try some more intensive tests later today that page through the full inventory.
    James Ratcliffe
    Lightspeed HQ
  • gregaricangregarican Member Posts: 708 
    edited September 2018
    That's definitely a relief! I just tried a test of pulling an offset upwards of 6,000 and all was well too. Didn't mean to hop on the thread, just was interested in any potential scalability issue down the road :smile:

    cache-control: no-cache
    Postman-Token: 907f0fed-8115-4148-9670-6d7629512854
    Authorization: Bearer {AccessToken}
    User-Agent: PostmanRuntime/7.1.1
    Accept: */*
    Host: api.merchantos.com
    cookie: __cfduid=db226f08af7bd70432115fe0872508a6f1518446657
    accept-encoding: gzip, deflate
    Connection: close


    HTTP/1.1 200 OK
    Date: Fri, 07 Sep 2018 15:32:11 GMT
    Content-Type: application/json
    Content-Length: 74018
    Connection: close
    x-frame-options: SAMEORIGIN
    X-XSS-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    X-LS-Acct-Id: XXXXX
    X-LS-OAuth-Client-Id: XXXXX
    X-LS-API-Bucket-Level: 1/90
    X-LS-Shard-Id: 14
    X-LS-API-Drip-Rate: 1
    X-LS-Master-System: true
    X-LS-Master-Account: true
    X-LS-Master-Catalog: false
    Vary: Accept-Encoding
    Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
    Server: cloudflare
    CF-RAY: 456a39fe6c9a25bb-ORD

    @{"@attributes":{"count":"6137","offset":"6000","limit":"100"},"Item":[{"itemID":"6003","systemSku":"210000006003","defaultCost":"0","avgCost":"0","discountable":"true","tax":"true","archived":"false","itemType":"default","serialized":"false","description":"Bridal 14k White Gold Diamond Semi-mount Ring","modelYear":"0","upc":"","ean":"","customSku":"","manufacturerSku":"Y8743AA","createTime":"2018-09-07T15:13:05+00:00","timeStamp":"2018-09-07T15:13:05+00:00","publishToEcom":"true","categoryID":"0","taxClassID":"0","departmentID":"0","itemMatrixID":"0","manufacturerID":"20","seasonID":"0","defaultVendorID":"38","Prices":{"ItemPrice":[{"amount":"1083.15","useTypeID":"1","useType":"Default"},{"amount":"1083.15","useTypeID":"2","useType":"MSRP"},{"amount":"0","useTypeID":"3","useType":"Online"}]}},
    Post edited by jamesratcliffe on
  • gregaricangregarican Member Posts: 708 
    edited September 2018
    Actually I might have spoken too soon. That query was against a test account, without custom fields. When I ran a query against our production account and included custom fields then my query timed out when I tried to pull starting at offset 2570. Details below.

    Since we utilize the custom fields for e-com integration this might be an issue, correct? Since the timeout appears to be coming from the LS Retail API back-end and not my client. Unless I'm missing something. Currently seeing API timeouts even when pulling any offsets lower than this now...

    cache-control: no-cache
    Postman-Token: f05a3f9a-1db3-43c3-898b-87d8f1cfdf0e
    Authorization: Bearer {AccessToken}
    User-Agent: PostmanRuntime/7.1.1
    Accept: */*
    Host: api.merchantos.com
    cookie: __cfduid=db226f08af7bd70432115fe0872508a6f1518446657
    accept-encoding: gzip, deflate
    Connection: close


    HTTP/1.1 500 Internal Server Error
    Date: Fri, 07 Sep 2018 16:02:35 GMT
    Content-Type: application/json
    Content-Length: 114
    Connection: close
    x-frame-options: SAMEORIGIN
    X-XSS-Protection: 1; mode=block
    X-Content-Type-Options: nosniff
    X-LS-Acct-Id: XXXXX
    X-LS-OAuth-Client-Id: XXXXX
    X-LS-API-Bucket-Level: 1/100
    X-LS-Shard-Id: 18
    X-LS-API-Drip-Rate: 5
    X-LS-Master-System: true
    X-LS-Master-Account: true
    X-LS-Master-Catalog: false
    Vary: Accept-Encoding
    Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
    Server: cloudflare
    CF-RAY: 456a659b48d99d91-ORD

    {"httpCode":"500","httpMessage":"Internal Server Error","message":"Maximum execution time of 30 seconds exceeded"}
    Post edited by jamesratcliffe on
  • gregaricangregarican Member Posts: 708 
    edited September 2018
    Of course the workaround for this apparent issue is pulling each item serially by its ItemId. Including custom fields and whatnot. At an API rate of 1 pull/second then 6,000 items would take almost 2 hours. I wish I would've known about the impact of custom fields when we were first testing LS Retail. And were laying out the structure of how we planned to operate. But I'm sure we here will figure something out :smiley:

    Pulling items with custom fields using the paginated method allows me a limit of ~80 items per page. That's not too bad. So if push came to shove I'd use that method versus the serialized one.
    Post edited by gregarican on
  • jamesratcliffejamesratcliffe Administrator, Lightspeed Staff Posts: 160 moderator
    @cralls I was able to loop through all your items in a PHP script. Some calls took longer than 10 seconds, so you will need to either raise your timeout or use a lower `limit` value.

    @gregarican The response times are much higher on your account. You must be using custom fields very extensively. I think the solutions is still to use a lower limit. You can either use a fixed limit that you think is safe or find a way to dynamically adjust the limit when calls are failing. In our experience 50 is a safe value.
    James Ratcliffe
    Lightspeed HQ
  • crallscralls Member Posts: 12
    So this is working now. Something had to change on that end though because I tried grabbing as little as 10 products at a time previously and it still timed out. Now I'm able to reduce the number of products, increase the timeout and it works? As long as it works I'm good but there's no explanation as to why which doesn't instill confidence but as long as it keeps working I think we're good.

    Also, we're running about 10 or so custom fields and only 3 or 4 are text fields with very little text in them. It doesn't seem to me that would make the process "intensive" but perhaps the system is optimized for the default product fields and is sensitive to adding any more data to the records?
  • gregaricangregarican Member Posts: 708 
    @cralls,  if it's some SQL-type back end then these queries that include load_relations are likely involving table joins. That should introduce processing delays to some degree. Yet I wouldn't imagine they would be to the degree we see. But I saw what you saw, in that I encountered timeouts during a certain period of time, but later on using the same exact API query I was able to pull the data just fine.

    @jamesratcliffe, we are frankly over-using the custom fields, but I unfortunately wasn't involved in the planning phase on our end. And if I would've been privy to the performance penalties they would introduce I definitely would've advised our team against over-reliance on them! 
  • jamesratcliffejamesratcliffe Administrator, Lightspeed Staff Posts: 160 moderator
    @cralls If performance on our servers was so bad that you couldn't get 10 products at a time (even on one shard), it would have affected a lot of customers, but we didn't have any other reports of slowness. I looked at our system monitoring logs, and everything looks normal last week.


    @gregarican What you saw seems is probably on our end. Response times do vary, so the same call could take, say 32 seconds at one point in the time, then 27 seconds later on when the system is under less strain. The 32-second call will fail with a timeout error (status 500), but the 27-second call will go through.


    We're working on improving how inventory is handled, so we'll hopefully see improvements to response times in the future and won't have to worry about these issues.
    James Ratcliffe
    Lightspeed HQ
  • crallscralls Member Posts: 12
    James, so I couldn't get 10 products at a time with a 10 second time out. But now I can get 50 at a time with a 30 second timeout so it doesn't add up but I'm just glad it's working now. Thanks for being so responsive!
  • jamesratcliffejamesratcliffe Administrator, Lightspeed Staff Posts: 160 moderator
    @cralls You're welcome.
    James Ratcliffe
    Lightspeed HQ
Sign In or Register to comment.