elasticsearch cluster red status fix due to high watermark

Do not delete elasticsearch files with rm on filesystem, use es api instead.

curl -X GET "localhost:9200/_cluster/health" | jq

or

curl -X GET -u elastic:YourGigaPassword "localhost:9200/_cluster/health" | jq

"status": "red"

List indicies sorted by size:

curl -X GET "localhost:9200/_cat/indices?s=store.size:asc" 

Delete index:

curl -X DELETE  "localhost:9200/.ds-logs-kubernetes.container_logs-default-2024.11.12-000002"

After that setup index lifecycle management.

 Links:

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete.html

https://www.elastic.co/guide/en/elasticsearch/reference/8.13/indices-delete-index.html

https://stackoverflow.com/questions/61327189/how-to-sort-kibana-indices-by-descending-size

https://stackoverflow.com/questions/24960902/how-to-get-a-list-of-elasticsearch-indices-sorted-by-name

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/cat-indices.html

https://www.elastic.co/guide/en/elasticsearch/reference/8.13/cat-indices.html