API Connection Timeout?

cralls

So I'm connecting to the Retail API using the Item endpoint to pull our catalog. I'm able to pull 100 products at a time but we get to a certain point and the request just times out. Are we hitting a rate limit? I'm not able to tell because there is no response from the API. How many requests per minute can we make? Is there a log somewhere on the lightspeed side that can give me more information about why exactly the API stops responding?

cralls

More info, looking at the API response I'm almost getting the entire catalog. My bucket is always at 59 so I don't think it's a rate limit issue but for some reason when I request products 7500 to 7600 the API times out and doesn't send me a response.

Array

(

    [count] => 7854

    [offset] => 7400

    [limit] => 100

    [free_bucket] => 59

)

jamesratcliffe

@cralls There's something about one or more of the items in that range that's making them take longer to load.

You might get the response if you raise your timeout limit. You can also try using a limit of 50.

cralls

That's what I figured. I need your help figuring out which item is causing the problem. I can't figure it out without some kind of a log or some kind of feedback from that end. All I'm getting is a time out. I've tried raising our timeout to 60 seconds between requests and still have the same issue so this isn't any kind of a rate-limiting issue.

gregarican

When I run into things like that I keep "cutting in half." Try pulling only records 7500 through 7550. If that works, then pull 7551 through 7600. Then once that fails (assuming it would) then pull 7551 through 7575. If that works then pull 7576 through 7600. And so on. Looking at maybe 5-6 iterations to figure out which record is causing an issue. 

Clunky, but if you are kind of working against a black box of sorts that's about the only suggestion I can think of.

jamesratcliffe

@cralls Send me the account ID in a private message; I can look into it.

In many cases, there isn't actually anything wrong with the item(s), they just have more data attached to them (usually related data, especially custom field values). In these cases, you just have to split that page of 100 into smaller chunks.

cralls

I try to limit of 50. Then I try to limit of 25. Still no luck. I actually didn't even make it as far as that same range when I did this.

gregarican

Can you run a Fiddler trace to see what's going on? That's the oddest thing. Do you have a lot of custom fields in the recordset? Also, is your code iterating through the paginated results? If so, perhaps just try curl'ing the specific offset and record count you are looking for. Never run into this on my end...

jamesratcliffe

I don't think I've seen a case where a call with limit=50 or less failed because of resource limitations.

Can you send me the account ID in a private message?

cralls

So it seems I cannot grab ANY products after 6000. I am iterating through 10 at a time at this point and when it gets to 6000 I get "Read timed out after 10 seconds" in my log i see the following:

[2018-09-06 20:21:54] main.CRITICAL: Read timed out after 10 seconds {"exception":"[object] (Zend_Http_Client_Adapter_Exception(code: 1000): Read timed out after 10 seconds at /home/discountgamesinc/public_html/vendor/magento/zendframework1/library/Zend/Http/Client/Adapter/Socket.php:512)"} []

If I start at 6000 and try to move forward from there it times out right away so basically it seems the API is limited to dealing with the first 6000 products in your catalog?

gregarican

Uh oh. Just reading this. I'll definitely keep an eye out on this. We are currently at around 4,300 products. But are looking to convert a few other of our companies. Which should put us closer to around 19-20K products when all is said and done. @Michael Carey looping you in since you are the expert on these matters

jamesratcliffe

@cralls I tried calls on your account with offsets greater than 6000 with no issues. The response time is pretty high when loading custom field values, though.

For example, this call took 13.261 seconds but was successful:

/API/Account/XXXX/Item.json?load_relations=["CustomFieldValues"]&offset=6100<br>

The response is faster with a smaller page size. This call took 5.750 seconds:

/API/Account/XXXX/Item.json?load_relations=["CustomFieldValues"]&offset=6100&limit=50<br>

It looks like your more recent items just have more custom field values set, so the calls are more intensive.


There are no limits in place related to the total number of items. In practice, you might see performance suffer in the frontend if you have a lot of items, but only as you start to approach 100k. 50k or less will be fine.


So far this doesn't look like an issue on our end, but I'll try some more intensive tests later today that page through the full inventory.

gregarican

That's definitely a relief! I just tried a test of pulling an offset upwards of 6,000 and all was well too. Didn't mean to hop on the thread, just was interested in any potential scalability issue down the road

GET https://api.merchantos.com/API/Account/{AccountId}/Item.json?offset=6000&limit=100 HTTP/1.1

cache-control: no-cache

Postman-Token: 907f0fed-8115-4148-9670-6d7629512854

Authorization: Bearer {AccessToken}

User-Agent: PostmanRuntime/7.1.1

Accept: */*

Host: api.merchantos.com

cookie: __cfduid=db226f08af7bd70432115fe0872508a6f1518446657

accept-encoding: gzip, deflate

Connection: close

HTTP/1.1 200 OK

Date: Fri, 07 Sep 2018 15:32:11 GMT

Content-Type: application/json

Content-Length: 74018

Connection: close

x-frame-options: SAMEORIGIN

X-XSS-Protection: 1; mode=block

X-Content-Type-Options: nosniff

X-LS-Acct-Id: XXXXX

X-LS-OAuth-Client-Id: XXXXX

X-LS-API-Bucket-Level: 1/90

X-LS-Shard-Id: 14

X-LS-API-Drip-Rate: 1

X-LS-Master-System: true

X-LS-Master-Account: true

X-LS-Master-Catalog: false

Vary: Accept-Encoding

Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"

Server: cloudflare

CF-RAY: 456a39fe6c9a25bb-ORD

@{"@attributes":{"count":"6137","offset":"6000","limit":"100"},"Item":[{"itemID":"6003","systemSku":"210000006003","defaultCost":"0","avgCost":"0","discountable":"true","tax":"true","archived":"false","itemType":"default","serialized":"false","description":"Bridal 14k White Gold Diamond Semi-mount Ring","modelYear":"0","upc":"","ean":"","customSku":"","manufacturerSku":"Y8743AA","createTime":"2018-09-07T15:13:05+00:00","timeStamp":"2018-09-07T15:13:05+00:00","publishToEcom":"true","categoryID":"0","taxClassID":"0","departmentID":"0","itemMatrixID":"0","manufacturerID":"20","seasonID":"0","defaultVendorID":"38","Prices":{"ItemPrice":[{"amount":"1083.15","useTypeID":"1","useType":"Default"},{"amount":"1083.15","useTypeID":"2","useType":"MSRP"},{"amount":"0","useTypeID":"3","useType":"Online"}]}},

gregarican

Actually I might have spoken too soon. That query was against a test account, without custom fields. When I ran a query against our production account and included custom fields then my query timed out when I tried to pull starting at offset 2570. Details below.

Since we utilize the custom fields for e-com integration this might be an issue, correct? Since the timeout appears to be coming from the LS Retail API back-end and not my client. Unless I'm missing something. Currently seeing API timeouts even when pulling any offsets lower than this now...

GET https://api.merchantos.com/API/Account/{AccountId}/Item.json?load_relations=["CustomFieldValues", "CustomFieldValues.value"]&offset=2570&limit=100 HTTP/1.1

cache-control: no-cache

Postman-Token: f05a3f9a-1db3-43c3-898b-87d8f1cfdf0e

Authorization: Bearer {AccessToken}

User-Agent: PostmanRuntime/7.1.1

Accept: */*

Host: api.merchantos.com

cookie: __cfduid=db226f08af7bd70432115fe0872508a6f1518446657

accept-encoding: gzip, deflate

Connection: close

HTTP/1.1 500 Internal Server Error

Date: Fri, 07 Sep 2018 16:02:35 GMT

Content-Type: application/json

Content-Length: 114

Connection: close

x-frame-options: SAMEORIGIN

X-XSS-Protection: 1; mode=block

X-Content-Type-Options: nosniff

X-LS-Acct-Id: XXXXX

X-LS-OAuth-Client-Id: XXXXX

X-LS-API-Bucket-Level: 1/100

X-LS-Shard-Id: 18

X-LS-API-Drip-Rate: 5

X-LS-Master-System: true

X-LS-Master-Account: true

X-LS-Master-Catalog: false

Vary: Accept-Encoding

Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"

Server: cloudflare

CF-RAY: 456a659b48d99d91-ORD

{"httpCode":"500","httpMessage":"Internal Server Error","message":"Maximum execution time of 30 seconds exceeded"}

gregarican

Of course the workaround for this apparent issue is pulling each item serially by its ItemId. Including custom fields and whatnot. At an API rate of 1 pull/second then 6,000 items would take almost 2 hours. I wish I would've known about the impact of custom fields when we were first testing LS Retail. And were laying out the structure of how we planned to operate. But I'm sure we here will figure something out

Pulling items with custom fields using the paginated method allows me a limit of ~80 items per page. That's not too bad. So if push came to shove I'd use that method versus the serialized one.

jamesratcliffe

@cralls I was able to loop through all your items in a PHP script. Some calls took longer than 10 seconds, so you will need to either raise your timeout or use a lower `limit` value.

@gregarican The response times are much higher on your account. You must be using custom fields very extensively. I think the solutions is still to use a lower limit. You can either use a fixed limit that you think is safe or find a way to dynamically adjust the limit when calls are failing. In our experience 50 is a safe value.

cralls

So this is working now. Something had to change on that end though because I tried grabbing as little as 10 products at a time previously and it still timed out. Now I'm able to reduce the number of products, increase the timeout and it works? As long as it works I'm good but there's no explanation as to why which doesn't instill confidence but as long as it keeps working I think we're good.

Also, we're running about 10 or so custom fields and only 3 or 4 are text fields with very little text in them. It doesn't seem to me that would make the process "intensive" but perhaps the system is optimized for the default product fields and is sensitive to adding any more data to the records?

gregarican

@cralls,  if it's some SQL-type back end then these queries that include load_relations are likely involving table joins. That should introduce processing delays to some degree. Yet I wouldn't imagine they would be to the degree we see. But I saw what you saw, in that I encountered timeouts during a certain period of time, but later on using the same exact API query I was able to pull the data just fine.

@jamesratcliffe, we are frankly over-using the custom fields, but I unfortunately wasn't involved in the planning phase on our end. And if I would've been privy to the performance penalties they would introduce I definitely would've advised our team against over-reliance on them! 

jamesratcliffe

@cralls If performance on our servers was so bad that you couldn't get 10 products at a time (even on one shard), it would have affected a lot of customers, but we didn't have any other reports of slowness. I looked at our system monitoring logs, and everything looks normal last week.


@gregarican What you saw seems is probably on our end. Response times do vary, so the same call could take, say 32 seconds at one point in the time, then 27 seconds later on when the system is under less strain. The 32-second call will fail with a timeout error (status 500), but the 27-second call will go through.


We're working on improving how inventory is handled, so we'll hopefully see improvements to response times in the future and won't have to worry about these issues.

cralls

James, so I couldn't get 10 products at a time with a 10 second time out. But now I can get 50 at a time with a 30 second timeout so it doesn't add up but I'm just glad it's working now. Thanks for being so responsive!

jamesratcliffe

@cralls You're welcome.