This blog post is a short example of connecting to Elasticsearch on IBM Cloud with Databases for Elasticsearch using bash scripting with cURL.
This GitHub project contains the source code, which is shown in this blog post: https://github.com/thomassuedbroecker/example-connect-to-elastic-search
Content of this blog post
- Examples that are used in the bash automation
- Setup
- Example usage of search indexes using bash automation
- Example single cURL command for a search in Elasticsearch
Useful resources:
- Elasticsearch API documentation here you can easily copy the commands
- IBM Cloud Elasticsearch
CURLdocumentation
1. Examples that are used in the bash automation
Here you see an extraction for each of the topics from the bash automation.
1.1. Create an index
Link to Elasticsearch help: Create index
INDEX_TO_CREATE=documents
echo "Create: $INDEX_TO_CREATE"
CURL_CA_BUNDLE=$E_CERT_PATH/$E_CERT_FILE_NAME curl -u ${E_ADMIN_USER}:${E_ADMIN_PASSWORD} -i -XPUT "https://${E_HOST}:${E_PORT}/$INDEX_TO_CREATE?pretty"
1.2. Search inside an index
Link to Elasticsearch help: Search index
CURL_CA_BUNDLE=$E_CERT_PATH/$E_CERT_FILE_NAME curl -u ${E_ADMIN_USER}:${E_ADMIN_PASSWORD} -XGET -H "Content-Type: application/json" "https://${E_HOST}:${E_PORT}/documents/_search" -d '{"query" : {"match_all" : {}}}' | jq '.'
1.3 Delete an index
Link to Elasticsearch help: Delete index
INDEX_TO_DELETE=documents
echo "Delete: $INDEX_TO_DELETE"
CURL_CA_BUNDLE=$E_CERT_PATH/$E_CERT_FILE_NAME curl -u ${E_ADMIN_USER}:${E_ADMIN_PASSWORD} -i -XDELETE "https://${E_HOST}:${E_PORT}/$INDEX_TO_DELETE?pretty"
1.4 Upload data
Link to Elasticsearch help: Upload documents
This example shows how to upload multiple documents and how to create an array of filenames in bash and loop a curl command.
function upload_index_documents () {
cd $HOME_PATH/full
directory="$(pwd)"
INDEX=documents
array=()
for file_name in $directory/*.json; do
if [ -f "$file_name" ]; then
data=$(echo $file_name | sed -r -e 's/([0-9]+)/ \1/' | sort -k 2 -n | sed -e 's/ //;')
array+=("$data")
fi
done
for file in "${array[@]}";do
echo "Upload next file: $file"
cat $file
cat "$file" | jq -c '.[] | {"index": {"_index": "documents", "_id": .url}}, .' | CURL_CA_BUNDLE=$E_CERT_PATH/$E_CERT_FILE_NAME curl -u ${E_ADMIN_USER}:${E_ADMIN_PASSWORD} -i -XPOST -H "Content-Type: application/json" "https://${E_HOST}:${E_PORT}/_bulk" --data-binary @-
done
cd $HOME_PATH
}
2. Setup
This is an example setup with bash automation you can follow along.
Step 1: Clone the repo to the local machine
git clone https://github.com/thomassuedbroecker/example-connect-to-elastic-search
cd example-connect-to-elastic-search
Step 2: Create a Standard Eleasticsearch instance on IBM Cloud
open https://cloud.ibm.com/databases/databases-for-elasticsearch/create
Step 3: Create new Service Credentials
Step 4: Download the self-signed certificate to your local computer
Download the certificate into the folder ./code/cert.
Step 5: Create a .env file for environment variables of the bash automation
cat code/.env_template > .env
Step 6: Insert the needed values for the environment variables
# IBM Cloud
export IBM_CLOUD_API_KEY=XXXX
export IBM_CLOUD_REGION=us-south
export IBM_CLOUD_RESOURCE_GROUP=default
# Elasticsearch service
export E_SEARCH_SERVICE=YOUR_DatabasesForElasticsearch
export E_CERT_FILE_NAME=d5290bfc-XXXXX-XXX-9337-XXXX40bd
export E_HOST=XXXXX-XXXXX-XXXXX-XXXXX.XXXX.databases.appdomain.cloud
export E_PORT=0815
export E_ADMIN_USER=admin
export E_ADMIN_PASSWORD=YOUR_AWESOME_PASSWORD
Step 7: Change the admin password for the database
cd /code
sh change_admin_password.sh
2. Example usage of search indexes using bash automation
These are the bash automation scripts.
Step 1: Test the connection
cd /code
sh test_connection.sh
Step 2: Create indexes
cd /code
sh create_indexes.sh
Step 3: Upload data to index
cd /code
sh upload-data_to_indexes.sh
Step 4: Search in index
cd /code
sh search_in_indexes.sh
Step 5: Delete indexes
cd /code
sh delete_indexes.sh
Elasticsearch example for a single CURL search command
Step 1: Open a terminal
Step 2: Create a folder called cert
mkdir cert
Step 3: Navigate to the folder
cd cert
Step 4: Set the environment variable path
export E_CERT_PATH=$(pwd)
Step 5: Download the certificate file for your Databases for Elasticsearch instance
Download it to your local computer into the newly created folder.
Step 5: Set the following environment variables to your values
- Example:
export E_CERT_FILE_NAME=d5290bfc-XXXXX-XXX-9337-XXXX40bd
export E_HOST=XXXXX-XXXXX-XXXXX-XXXXX.XXXX.databases.appdomain.cloud
export E_PORT=0815
export E_ADMIN_USER=admin
export E_ADMIN_PASSWORD=YOUR_AWESOME_PASSWORD
Step 6: Execute an index search
CURL_CA_BUNDLE=$E_CERT_PATH/$E_CERT_FILE_NAME curl -u ${E_ADMIN_USER}:${E_ADMIN_PASSWORD} -XGET -H "Content-Type: application/json" "https://${E_HOST}:${E_PORT}/documents/_search" -d '{ "query": {"multi_match" : {"query" : "Second Hello World?","fields": ["title", "text"]}}}' | jq '.'
I hope this was useful to you, and let’s see what’s next?
Greetings,
Thomas
#curl, #bashscript, #api, #elasticsearch, #ibmcloud
Thomas Suedbroecker's Blog 
Leave a Reply