Notifications
Clear all
Topic starter
Hi there. For several days, I've been attempting to retrieve batch information from a database and am having significant difficulty consolidating the outputs into a single list for subsequent processing. I've reached the stage of storing them in a PostgreSQL table, but then the workflow progresses about a third of the way before abruptly stopping (completing). There are over 3300 records in total. Could someone please advise on where I might be going wrong? Thank you!
Posted : 04/04/2025 2:03 pm
Hello! Could you share the workflow's JSON and your callin.io version, please?
Posted : 04/04/2025 2:37 pm
For me, when this occurs, it's typically because some data didn't pass through during the run. It's crucial to check the executions tab to see where it failed and what the data looked like.
Here's my current troubleshooting approach:
- Executions Tab: Was the data missing or improperly formatted?
- In several instances, the data was simply EMPTY. I had to adjust my node to ALWAYS produce an output. You can do this by navigating into the node's settings and ensuring it always generates an output, even if it's blank. This, of course, depends on your specific workflow and desired outcome. In some scenarios, you might actually want it to fail on empty data to help identify which data is returning a null value.
Once I identify the problematic data, I either edit the data itself OR modify the workflow to accommodate it.
Take a look and see.
Posted : 04/04/2025 2:41 pm
Topic starter
Hi. The current version is 1.80.3, self-hosted. Please see the JSON below:
Posted : 04/04/2025 2:46 pm
It might be a timeout issue, especially if you don't observe it failing on specific data. In such scenarios, I've found that incorporating a WAIT node could resolve the problem.
If anyone else has suggestions, please feel free to share. I suspect this could be the cause if it's a connection-related issue (using a WAIT node to allow sufficient time for database loading).
Additionally, I recommend checking the executions list to pinpoint where the failure occurred. This typically provides valuable insight.
Posted : 04/04/2025 3:02 pm
Hi there, I'm encountering an issue when attempting to copy and paste the workflow. Could you please use triple backticks (
```
) when sharing your workflow? This will ensure it's displayed correctly. Thanks!
Posted : 04/04/2025 3:08 pm
Topic starter
"name": "Store all BMS Service Items",
"nodes": [
{
"parameters": {},
"type": "n8n-nodes-base.manualTrigger",
"typeVersion": 1,
"position": [
140,
0
],
"id": "1169d60a-2c5c-4fbd-a999-294c8a34e04f",
"name": "When clicking ‘Test workflow’"
},
{
"parameters": {
"method": "POST",
"url": "https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/sessions",
"authentication": "genericCredentialType",
"genericAuthType": "httpBasicAuth",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
}
]
},
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
380,
0
],
"id": "8cf8a0ec-1856-469a-b565-03b29189af19",
"name": "HTTP Request",
"credentials": {
"httpHeaderAuth": {
"id": "SO772OpPzuKqORgc",
"name": "Header Auth Filemaker"
},
"httpCustomAuth": {
"id": "RrxkUhkNKPCb6JyJ",
"name": "Filemaker"
},
"httpBasicAuth": {
"id": "dsJ5HTRAU8uMbTB3",
"name": "Filemaker"
}
}
},
{
"parameters": {
"method": "POST",
"url": "https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/layouts/Services_Detail/_find",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
},
{
"name": "Authorization",
"value": "=Bearer {{ $('HTTP Request').item.json.response.token }}"
}
]
},
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={{ $json}}",
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
1000,
0
],
"id": "bf9a8d0c-1dcf-4832-a10d-8d07bab48ad5",
"name": "HTTP Request1"
},
{
"parameters": {
"modelId": {
"__rl": true,
"value": "gpt-4o",
"mode": "list",
"cachedResultName": "GPT-4O"
},
"messages": {
"values": [
{
"content": "=IMPORTANT: Use only UK-English, NOT US-English. nnYour role is to create a summarised datasheet on an attestation service offered by the company. Each service can differ on the basis of what document type it is, which country it was issued in, and which country it needs to be used in. Some services are further broken down by speed and/or variation of the document type and/or processing method. nnYou will be provided with a set of fields contained within the company's internal database. From these fields you should summarise, losing none of the actual detail, the service code and title, what the service is, how it is processed, and a range of costs based on the number of documents the client wishes to process. If quantity prices are not provided, just output the single fixed cost price. We also offer special rates for UK-issued documents (only) to registered corporate clients but these are by request. Please bear in mind that these prices exclude delivery and tax where appropriate. nnHere are the fields and details: nnService Code: {{ $json.fieldData.ServiceCode }}nnDescription: {{ $json.fieldData.Description }}nnIssuing Country: {{ $json.fieldData['Issue Country'] }}nnDestination Country: {{ $json.fieldData['Destination Country'] }}nnDocument Type: {{ $json.fieldData['Document Type'] }}nnService Variation: {{ $json.fieldData.ServiceVariant }}nnClient Advice Notes: {{ $json.fieldData.CommonClientAdvice }}nnClient Documents Required (If any): {{ $json.fieldData.ClientDocsRequired }}nnSpecific Service Notes: {{ $json.fieldData.ServiceNotes }}nnExpected Processing Time (not including shipping to/from our offices): n{{ $json.fieldData.UKTimeframe }}nnPrices in GBP:nn1 document: {{ $json.fieldData.ukSellPrice_1 }}nn2 documents: {{ $json.fieldData.ukSellPrice_2 }}nn3 Documents: {{ $json.fieldData.ukSellPrice_3 }}nn4 Documents: {{ $json.fieldData.ukSellPrice_4 }}nn5 or more documents: {{ $json.fieldData.ukSellPrice_5 }}nnFixed Price: {{ $json.fieldData.c_OwnSell }}n"
}
]
},
"options": {
"temperature": 0.5,
"maxToolsIterations": 0
}
},
"type": "@n8n/n8n-nodes-langchain.openAi",
"typeVersion": 1.8,
"position": [
3140,
260
],
"id": "18b8fa49-37be-4300-9458-d0f70b887949",
"name": "OpenAI",
"credentials": {
"openAiApi": {
"id": "bEeQ4sZ6w94tqHCT",
"name": "OpenAi account"
}
}
},
{
"parameters": {
"method": "DELETE",
"url": "= https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/sessions/{{ $('HTTP Request').item.json.response.token }}",
"authentication": "genericCredentialType",
"genericAuthType": "httpBasicAuth",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
}
]
},
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
2140,
20
],
"id": "afd997e9-a3f3-4540-b3bb-01274b6bc618",
"name": "HTTP Request3",
"credentials": {
"httpBasicAuth": {
"id": "dsJ5HTRAU8uMbTB3",
"name": "Filemaker"
}
}
},
{
"parameters": {
"method": "PATCH",
"url": "https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/globals",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
},
{
"name": "Authorization",
"value": "=Bearer {{ $json.response.token }}"
}
]
},
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={"globalFields":{"Services::zz_BranchID": 1}}",
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
600,
0
],
"id": "9286f0d7-374e-4420-b3ab-72867aeb4675",
"name": "HTTP Request4"
},
{
"parameters": {
"mode": "insert",
"pineconeIndex": {
"__rl": true,
"value": "vitalconsular",
"mode": "list",
"cachedResultName": "vitalconsular"
},
"options": {
"pineconeNamespace": "vital_bms_uk"
}
},
"type": "@n8n/n8n-nodes-langchain.vectorStorePinecone",
"typeVersion": 1,
"position": [
3780,
260
],
"id": "59c1347f-a1bc-400b-b1ce-67f7181afdf9",
"name": "Pinecone Vector Store",
"credentials": {
"pineconeApi": {
"id": "qKnHWvIBJLCYAuBt",
"name": "PineconeApi account"
}
}
},
{
"parameters": {
"options": {}
},
"type": "@n8n/n8n-nodes-langchain.embeddingsOpenAi",
"typeVersion": 1.2,
"position": [
3700,
540
],
"id": "95bb79b7-fca8-4295-b16a-0479ec89b799",
"name": "Embeddings OpenAI",
"credentials": {
"openAiApi": {
"id": "bEeQ4sZ6w94tqHCT",
"name": "OpenAi account"
}
}
},
{
"parameters": {
"jsonMode": "expressionData",
"jsonData": "=Service Code: {{ $('Split Out').item.json.fieldData.ServiceCode }}. This describes the {{ $('Split Out').item.json.fieldData.Description }} service.nn{{ $('OpenAI').item.json.message.content }}",
"options": {
"pointers": "={{ $json["text"] }}_chunk_{{ $itemIndex }}",
"metadata": {
"metadataValues": [
{
"name": "title",
"value": "={{ $('Split Out').item.json.fieldData.Description }}"
},
{
"name": "id",
"value": "={{ $('Split Out').item.json.fieldData.__Pk_Service }}"
},
{
"name": "service_code",
"value": "={{ $('Split Out').item.json.fieldData.ServiceCode }}"
},
{
"name": "issue_country",
"value": "={{ $('Split Out').item.json.fieldData["Issue Country"] }}"
},
{
"name": "destination_country",
"value": "={{ $('Split Out').item.json.fieldData["Destination Country"] }}"
},
{
"name": "document_category",
"value": "={{ $('Split Out').item.json.fieldData["Document Type"] }}"
}
]
}
}
},
"type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
"typeVersion": 1,
"position": [
3840,
540
],
"id": "6c3be04e-ab5f-4c0b-b1d3-e36cd275b7c4",
"name": "Default Data Loader"
},
{
"parameters": {
"chunkOverlap": 200,
"options": {}
},
"type": "@n8n/n8n-nodes-langchain.textSplitterRecursiveCharacterTextSplitter",
"typeVersion": 1,
"position": [
3840,
720
],
"id": "d5bbb5f4-9855-4610-85b9-15ae73801d4c",
"name": "Recursive Character Text Splitter"
},
{
"parameters": {
"fieldToSplitOut": "message.content",
"options": {}
},
"type": "n8n-nodes-base.splitOut",
"typeVersion": 1,
"position": [
3500,
260
],
"id": "5f3b5ec5-1bc5-4219-8621-23feca38b023",
"name": "Split Out1"
},
{
"parameters": {
"jsCode": "// Determine if this is the first iteration or part of a loopn// Default to 100 for the first batch after HTTP Request1nlet currentOffset = $json.offset || 100;nn// Build the FileMaker query with proper paginationnconst queryBody = {n query: [n { "service_PARTNERS::PartnerStatus": "Active" }n ],n limit: 100,n offset: currentOffsetn};nn// Log the current offset for debuggingnconsole.log(` Using offset: ${currentOffset}`);nn// Return the query for the HTTP Request2 nodenreturn {n json: {n body: queryBody, // Required for FileMaker Data APIn offset: currentOffset // Pass through for referencen }n};"
},
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
1300,
0
],
"id": "3d60445e-4c13-42da-8c0f-96d306a3f069",
"name": "Code"
},
{
"parameters": {
"jsCode": "// Retrieve the current offset (default to 0)nlet offset = $json.offset || 0;nn// Build the basic JSON body with your query and limitnlet body = {n query: [n { "service_PARTNERS::PartnerStatus": "Active" }n ],n limit: 100n}nnreturn { json: body };"
},
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
820,
0
],
"id": "3053f89a-6f98-43f0-837a-c902ab34a522",
"name": "Code1"
},
{
"parameters": {
"conditions": {
"options": {
"caseSensitive": true,
"leftValue": "",
"typeValidation": "strict",
"version": 2
},
"conditions": [
{
"id": "0edbabce-ea99-4570-918c-728c2ef6e4fa",
"leftValue": "={{ $json.continueLoop }}",
"rightValue": "true",
"operator": {
"type": "boolean",
"operation": "true",
"singleValue": true
}
}
],
"combinator": "and"
},
"options": {}
},
"type": "n8n-nodes-base.if",
"typeVersion": 2.2,
"position": [
1900,
0
],
"id": "a5990fd6-63aa-469e-8763-f3cd13b69c17",
"name": "If"
},
{
"parameters": {
"method": "POST",
"url": "https://fmaws.vitalcertificates.co.uk/fmi/data/v1/databases/VC Branch Management/layouts/Services_Detail/_find",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "Content-Type",
"value": "application/json"
},
{
"name": "Authorization",
"value": "=Bearer {{ $('HTTP Request').item.json.response.token }}"
}
]
},
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={{ $json.body }}",
"options": {}
},
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
1480,
0
],
"id": "18985872-170c-45c9-913c-7158739b861b",
"name": "HTTP Request2"
},
{
"parameters": {
"jsCode": "// Get the response data from HTTP Request2nconst response = $json.response;nconst dataInfo = response.dataInfo || {};nconst foundCount = dataInfo.foundCount || 0;nconst returnedCount = dataInfo.returnedCount || 0;nn// Get the current offset directly from the Code noden// This is the offset value that was used in the current requestnlet currentOffset;ntry {n currentOffset = $('Code').first().json.offset;n} catch (error) {n currentOffset = 100; // Default fallbackn console.log(`[Code2] Error accessing Code node: ${error.message}`);n}nn// Calculate the next offset for the following iterationnconst nextOffset = currentOffset + returnedCount;nn// Determine if we should continue the loopnconst continueLoop = nextOffset < foundCount;nn// Log information for debuggingnconsole.log(`[Code2] Current offset: ${currentOffset}, Records returned: ${returnedCount}`);nconsole.log(`[Code2] Next offset: ${nextOffset}, Total records: ${foundCount}`);nconsole.log(`[Code2] Continue loop: ${continueLoop}`);nn// Return data for the IF nodenreturn {n json: {n continueLoop: continueLoop, // Boolean for the IF node to evaluaten offset: nextOffset, // For the next iteration if loop continuesn n // Additional information for reference/debuggingn foundCount: foundCount,n returnedCount: returnedCountn }n};"
},
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
1740,
0
],
"id": "35b1ea5f-b22e-472f-a61d-8d8bd9487ceb",
"name": "Code2"
},
{
"parameters": {
"options": {}
},
"type": "n8n-nodes-base.splitInBatches",
"typeVersion": 3,
"position": [
2880,
260
],
"id": "34a7805a-f9b2-4b40-a6d7-253aa2936dde",
"name": "Loop Over Items"
},
{
"parameters": {},
"type": "n8n-nodes-base.merge",
"typeVersion": 3,
"position": [
1700,
260
],
"id": "b643271b-556d-4ef7-863d-784d869797c4",
"name": "Merge"
},
{
"parameters": {
"fieldToSplitOut": "response.data",
"options": {}
},
"type": "n8n-nodes-base.splitOut",
"typeVersion": 1,
"position": [
1900,
260
],
"id": "dfae9c84-287c-4ea7-84b2-7ea0b0ae6ce3",
"name": "Split Out2"
},
{
"parameters": {
"operation": "upsert",
"schema": {
"__rl": true,
"mode": "list",
"value": "public"
},
"table": {
"__rl": true,
"value": "data_collection",
"mode": "list",
"cachedResultName": "data_collection"
},
"columns": {
"mappingMode": "defineBelow",
"value": {
"id": "={{ $json.__Pk_Service }}",
"item_data": "={{ $json.item_data }}"
},
"matchingColumns": [
"id"
],
"schema": [
{
"id": "id",
"displayName": "id",
"required": false,
"defaultMatch": true,
"display": true,
"type": "number",
"canBeUsedToMatch": true
},
{
"id": "item_data",
"displayName": "item_data",
"required": false,
"defaultMatch": false,
"display": true,
"type": "object",
"canBeUsedToMatch": false
},
{
"id": "batch_id",
"displayName": "batch_id",
"required": false,
"defaultMatch": false,
"display": true,
"type": "string",
"canBeUsedToMatch": false
},
{
"id": "created_at",
"displayName": "created_at",
"required": false,
"defaultMatch": false,
"display": true,
"type": "dateTime",
"canBeUsedToMatch": false
}
],
"attemptToConvertTypes": false,
"convertFieldsToString": false
},
"options": {
"connectionTimeout": 300
}
},
"type": "n8n-nodes-base.postgres",
"typeVersion": 2.5,
"position": [
2420,
260
],
"id": "55cd32f0-b199-46bb-aad2-be03f96a75c9",
"name": "Postgres",
"credentials": {
"postgres": {
"id": "6pAW0I8XzLoTwtqS",
"name": "Postgres account"
}
}
},
{
"parameters": {
"operation": "select",
"schema": {
"__rl": true,
"mode": "list",
"value": "public"
},
"table": {
"__rl": true,
"value": "data_collection",
"mode": "list",
"cachedResultName": "data_collection"
},
"returnAll": true,
"options": {}
},
"type": "n8n-nodes-base.postgres",
"typeVersion": 2.5,
"position": [
2660,
-120
],
"id": "de818b9e-975f-4ca1-8fd9-8c63c102d273",
"name": "Postgres1",
"alwaysOutputData": false,
"credentials": {
"postgres": {
"id": "6pAW0I8XzLoTwtqS",
"name": "Postgres account"
}
}
},
{
"parameters": {
"mode": "runOnceForEachItem",
"jsCode": "const fieldData = $input.item.json.fieldData;nconst jsonString = JSON.stringify(fieldData);nconst base64String = Buffer.from(jsonString).toString('base64');nn// Extract the primary key fieldnconst primaryKeyService = fieldData.__Pk_Service || null;nnreturn {n json: {n // For PostgreSQL JSON field, we need to create a proper JSON objectn item_data: {n encoded_data: base64String,n primary_key: primaryKeyService,n encoding_type: "base64",n content_type: "application/json",n created_at: new Date().toISOString()n },n // Keep these separate for other uses if neededn __Pk_Service: primaryKeyService,n base64EncodedData: base64Stringn }n}"
},
"type": "n8n-nodes-base.code",
"typeVersion": 2,
"position": [
2160,
260
],
"id": "df29c1e0-dca2-426c-995a-a8fe036818ef",
"name": "Code4"
}
],
"pinData": {
"When clicking ‘Test workflow’": [
{
"json": {}
}
]
},
"connections": {
"When clicking ‘Test workflow’": {
"main": [
[
{
"node": "HTTP Request",
"type": "main",
"index": 0
}
]
]
},
"HTTP Request": {
"main": [
[
{
"node": "HTTP Request4",
"type": "main",
"index": 0
}
]
]
},
"HTTP Request1": {
"main": [
[
{
"node": "Code",
"type": "main",
"index": 0
},
{
"node": "Merge",
"type": "main",
"index": 1
}
]
]
},
"OpenAI": {
"main": [
[
{
"node": "Split Out1",
"type": "main",
"index": 0
}
]
]
},
"HTTP Request3": {
"main": [
[]
]
},
"HTTP Request4": {
"main": [
[
{
"node": "Code1",
"type": "main",
"index": 0
}
]
]
},
"Embeddings OpenAI": {
"ai_embedding": [
[
{
"node": "Pinecone Vector Store",
"type": "ai_embedding",
"index": 0
}
]
]
},
"Default Data Loader": {
"ai_document": [
[
{
"node": "Pinecone Vector Store",
"type": "ai_document",
"index": 0
}
]
]
},
"Recursive Character Text Splitter": {
"ai_textSplitter": [
[
{
"node": "Default Data Loader",
"type": "ai_textSplitter",
"index": 0
}
]
]
},
"Split Out1": {
"main": [
[
{
"node": "Pinecone Vector Store",
"type": "main",
"index": 0
}
]
]
},
"Code": {
"main": [
[
{
"node": "HTTP Request2",
"type": "main",
"index": 0
}
]
]
},
"Code1": {
"main": [
[
{
"node": "HTTP Request1",
"type": "main",
"index": 0
}
]
]
},
"If": {
"main": [
[
{
"node": "Code",
"type": "main",
"index": 0
}
],
[
{
"node": "HTTP Request3",
"type": "main",
"index": 0
}
]
]
},
"HTTP Request2": {
"main": [
[
{
"node": "Code2",
"type": "main",
"index": 0
},
{
"node": "Merge",
"type": "main",
"index": 0
}
]
]
},
"Code2": {
"main": [
[
{
"node": "If",
"type": "main",
"index": 0
}
]
]
},
"Pinecone Vector Store": {
"main": [
[
{
"node": "Loop Over Items",
"type": "main",
"index": 0
}
]
]
},
"Loop Over Items": {
"main": [
[],
[
{
"node": "OpenAI",
"type": "main",
"index": 0
}
]
]
},
"Merge": {
"main": [
[
{
"node": "Split Out2",
"type": "main",
"index": 0
}
]
]
},
"Split Out2": {
"main": [
[
{
"node": "Code4",
"type": "main",
"index": 0
}
]
]
},
"Code4": {
"main": [
[
{
"node": "Postgres",
"type": "main",
"index": 0
}
]
]
}
},
"active": false,
"settings": {
"executionOrder": "v1",
"callerPolicy": "workflowsFromSameOwner"
},
"versionId": "fb60dc1d-34ec-4185-b2af-4697d082aabb",
"meta": {
"templateCredsSetupCompleted": true,
"instanceId": "ae089b2b3e7b27ef840db210c133514d8bab95d260e915570f807f9e4e7bc7bf"
},
"id": "JlTztx5jb3CKKcxM",
"tags": []
}
Posted : 04/04/2025 3:12 pm
Topic starter
Yes, that seemed like a sensible suggestion, but it didn't result in any changes. I'm still encountering the 'connection lost' message.
Posted : 04/04/2025 3:14 pm
Topic starter
Just to clarify, the PostgreSQL approach is merely one of multiple solutions. I need to batch the data received from the HTTP modules because of record limitations, but then these batches become too large to process with an OpenAI node afterward (I'm attempting to populate a RAG with our datastore). I experimented with the loop method to segment these, but it only functions for the initial batch. This is proving to be quite challenging.
Posted : 04/04/2025 3:17 pm
Topic starter
Is that better?
Posted : 07/04/2025 3:18 pm
Topic starter
Posted : 07/04/2025 3:20 pm
This discussion was automatically closed 90 days following the last response. New responses are no longer permitted.
Posted : 06/07/2025 3:21 pm