API Connection Timeout?

in Development
So I'm connecting to the Retail API using the Item endpoint to pull our catalog. I'm able to pull 100 products at a time but we get to a certain point and the request just times out. Are we hitting a rate limit? I'm not able to tell because there is no response from the API. How many requests per minute can we make? Is there a log somewhere on the lightspeed side that can give me more information about why exactly the API stops responding?
20 comments
You might get the response if you raise your timeout limit. You can also try using a limit of 50.
Lightspeed HQ
Clunky, but if you are kind of working against a black box of sorts that's about the only suggestion I can think of.
In many cases, there isn't actually anything wrong with the item(s), they just have more data attached to them (usually related data, especially custom field values). In these cases, you just have to split that page of 100 into smaller chunks.
Lightspeed HQ
Can you send me the account ID in a private message?
Lightspeed HQ
[2018-09-06 20:21:54] main.CRITICAL: Read timed out after 10 seconds {"exception":"[object] (Zend_Http_Client_Adapter_Exception(code: 1000): Read timed out after 10 seconds at /home/discountgamesinc/public_html/vendor/magento/zendframework1/library/Zend/Http/Client/Adapter/Socket.php:512)"} []
If I start at 6000 and try to move forward from there it times out right away so basically it seems the API is limited to dealing with the first 6000 products in your catalog?
For example, this call took 13.261 seconds but was successful:
The response is faster with a smaller page size. This call took 5.750 seconds:
It looks like your more recent items just have more custom field values set, so the calls are more intensive.
There are no limits in place related to the total number of items. In practice, you might see performance suffer in the frontend if you have a lot of items, but only as you start to approach 100k. 50k or less will be fine.
So far this doesn't look like an issue on our end, but I'll try some more intensive tests later today that page through the full inventory.
Lightspeed HQ
Since we utilize the custom fields for e-com integration this might be an issue, correct? Since the timeout appears to be coming from the LS Retail API back-end and not my client. Unless I'm missing something. Currently seeing API timeouts even when pulling any offsets lower than this now...
Pulling items with custom fields using the paginated method allows me a limit of ~80 items per page. That's not too bad. So if push came to shove I'd use that method versus the serialized one.
@gregarican The response times are much higher on your account. You must be using custom fields very extensively. I think the solutions is still to use a lower limit. You can either use a fixed limit that you think is safe or find a way to dynamically adjust the limit when calls are failing. In our experience 50 is a safe value.
Lightspeed HQ
Also, we're running about 10 or so custom fields and only 3 or 4 are text fields with very little text in them. It doesn't seem to me that would make the process "intensive" but perhaps the system is optimized for the default product fields and is sensitive to adding any more data to the records?
@jamesratcliffe, we are frankly over-using the custom fields, but I unfortunately wasn't involved in the planning phase on our end. And if I would've been privy to the performance penalties they would introduce I definitely would've advised our team against over-reliance on them!
@gregarican What you saw seems is probably on our end. Response times do vary, so the same call could take, say 32 seconds at one point in the time, then 27 seconds later on when the system is under less strain. The 32-second call will fail with a timeout error (status 500), but the 27-second call will go through.
We're working on improving how inventory is handled, so we'll hopefully see improvements to response times in the future and won't have to worry about these issues.
Lightspeed HQ
Lightspeed HQ