Skip to content

ELK index_not_found_exception

🏷️ ELK

ELKlogstash 一般都是以日期为单位保存日志,格式如下:

groovy
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

通过代码查询日志时,需要指定索引,不过没有找到像 index pattern 一样支持通配符匹配索引的 API,于是只好自己根据系统时间计算索引名。

java
SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy.MM.dd");
String indexName = String.format("%s%s", indexPrefix, simpleDateFormat.format(new Date()));

执行后在 12 点钟的时候报了如下错误:

type=index_not_found_exception, reason=no such index

怀疑是时区导致的,查了一下,果然如此。每天 8 天以前的数据是保存在前一天日期对应的索引中的。
找到问题就很好解决了,指定 SimpleDateFormat 的时区之后再执行格式化就可以了。

java
SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy.MM.dd");
simpleDateFormat.setTimeZone(TimeZone.getTimeZone("Etc/GMT+0"));
String indexName = String.format("%s%s", indexPrefix, simpleDateFormat.format(new Date()));

附 1. 查询 ES 错误日志数量的代码

java
import cn.hutool.core.date.DateUtil;
import me.liujiajia.job.entity.log.LogWarningParameter;
import me.liujiajia.job.service.LogMessageService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang.StringUtils;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.core.CountRequest;
import org.elasticsearch.client.core.CountResponse;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.springframework.stereotype.Service;

import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.*;

@RequiredArgsConstructor
@Service
@Slf4j
public class LogMessageServiceImpl implements LogMessageService {

    private final RestHighLevelClient client;

    @Override
    public long logCount(LogWarningParameter parameter) throws IOException {
        if (parameter == null) return 0;

        String[] indices = getIndices(parameter.getIndexPrefix(), parameter.getMinutes());
        if (indices.length == 0) return 0;

        BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery();
        if (StringUtils.isNotEmpty(parameter.getLevel())) {
            queryBuilder.must(QueryBuilders.matchQuery("level", "ERROR"));
        }
        if (parameter.getMinutes() > 0) {
            queryBuilder.must(QueryBuilders.rangeQuery("@timestamp")
                    .from(String.format("now-%dm", parameter.getMinutes()), true)
                    .to("now", true));
        }
        if (parameter.getKeywords() != null && parameter.getKeywords().size() > 0) {
            parameter.getKeywords().forEach(keyword -> queryBuilder.must(QueryBuilders.fuzzyQuery("message", keyword)));
        }

        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(queryBuilder);

        CountRequest countRequest = new CountRequest(indices);
        countRequest.source(searchSourceBuilder);

        CountResponse countResponse = client.count(countRequest, RequestOptions.DEFAULT);
        return countResponse == null ? 0 : countResponse.getCount();
    }

    private String[] getIndices(String indexPrefix, int minutes) {
        Date now = new Date();
        Date fromTime = DateUtil.offsetMinute(now, -minutes);

        Set<String> result = new HashSet<>();
        SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy.MM.dd");
        simpleDateFormat.setTimeZone(TimeZone.getTimeZone("Etc/GMT+0"));
        while (fromTime.compareTo(now) < 0) {
            result.add(String.format("%s%s", indexPrefix, simpleDateFormat.format(fromTime)));
            fromTime = DateUtil.offsetDay(fromTime, 1);
        }
        result.add(String.format("%s%s", indexPrefix, simpleDateFormat.format(now)));

        return result.toArray(new String[result.size()]);
    }
}

附 2. 可用 TimeZone 列表

摘自 Joda-Time (2021-03-17) .

Standard OffsetCanonical IDAliases
-12:00Etc/GMT+12
-11:00Etc/GMT+11
-11:00Pacific/Midway
-11:00Pacific/Niue
-11:00Pacific/Pago_PagoPacific/Samoa, US/Samoa
-10:00America/AdakAmerica/Atka, US/Aleutian
-10:00Etc/GMT+10
-10:00HST
-10:00Pacific/HonoluluPacific/Johnston, US/Hawaii
-10:00Pacific/Rarotonga
-10:00Pacific/Tahiti
-09:30Pacific/Marquesas
-09:00America/AnchorageUS/Alaska
-09:00America/Juneau
-09:00America/Metlakatla
-09:00America/Nome
-09:00America/Sitka
-09:00America/Yakutat
-09:00Etc/GMT+9
-09:00Pacific/Gambier
-08:00America/Dawson
-08:00America/Los_AngelesUS/Pacific, US/Pacific-New
-08:00America/TijuanaAmerica/Ensenada, America/Santa_Isabel, Mexico/BajaNorte
-08:00America/VancouverCanada/Pacific
-08:00America/WhitehorseCanada/Yukon
-08:00Etc/GMT+8
-08:00PST8PDT
-08:00Pacific/Pitcairn
-07:00America/Boise
-07:00America/Cambridge_Bay
-07:00America/Chihuahua
-07:00America/Creston
-07:00America/Dawson_Creek
-07:00America/DenverAmerica/Shiprock, Navajo, US/Mountain
-07:00America/EdmontonCanada/Mountain
-07:00America/Fort_Nelson
-07:00America/Hermosillo
-07:00America/Inuvik
-07:00America/MazatlanMexico/BajaSur
-07:00America/Ojinaga
-07:00America/PhoenixUS/Arizona
-07:00America/Yellowknife
-07:00Etc/GMT+7
-07:00MST
-07:00MST7MDT
-06:00America/Bahia_Banderas
-06:00America/Belize
-06:00America/ChicagoUS/Central
-06:00America/Costa_Rica
-06:00America/El_Salvador
-06:00America/Guatemala
-06:00America/Indiana/KnoxAmerica/Knox_IN, US/Indiana-Starke
-06:00America/Indiana/Tell_City
-06:00America/Managua
-06:00America/Matamoros
-06:00America/Menominee
-06:00America/Merida
-06:00America/Mexico_CityMexico/General
-06:00America/Monterrey
-06:00America/North_Dakota/Beulah
-06:00America/North_Dakota/Center
-06:00America/North_Dakota/New_Salem
-06:00America/Rainy_River
-06:00America/Rankin_Inlet
-06:00America/ReginaCanada/Saskatchewan
-06:00America/Resolute
-06:00America/Swift_Current
-06:00America/Tegucigalpa
-06:00America/WinnipegCanada/Central
-06:00CST6CDT
-06:00Etc/GMT+6
-06:00Pacific/EasterChile/EasterIsland
-06:00Pacific/Galapagos
-05:00America/AtikokanAmerica/Coral_Harbour
-05:00America/Bogota
-05:00America/Cancun
-05:00America/Cayman
-05:00America/DetroitUS/Michigan
-05:00America/Eirunepe
-05:00America/Grand_Turk
-05:00America/Guayaquil
-05:00America/HavanaCuba
-05:00America/Indiana/IndianapolisAmerica/Fort_Wayne, America/Indianapolis, US/East-Indiana
-05:00America/Indiana/Marengo
-05:00America/Indiana/Petersburg
-05:00America/Indiana/Vevay
-05:00America/Indiana/Vincennes
-05:00America/Indiana/Winamac
-05:00America/Iqaluit
-05:00America/JamaicaJamaica
-05:00America/Kentucky/LouisvilleAmerica/Louisville
-05:00America/Kentucky/Monticello
-05:00America/Lima
-05:00America/Nassau
-05:00America/New_YorkUS/Eastern
-05:00America/Nipigon
-05:00America/Panama
-05:00America/Pangnirtung
-05:00America/Port-au-Prince
-05:00America/Rio_BrancoAmerica/Porto_Acre, Brazil/Acre
-05:00America/Thunder_Bay
-05:00America/TorontoAmerica/Montreal, Canada/Eastern
-05:00EST
-05:00EST5EDT
-05:00Etc/GMT+5
-04:00America/Anguilla
-04:00America/Antigua
-04:00America/Aruba
-04:00America/Asuncion
-04:00America/Barbados
-04:00America/Blanc-Sablon
-04:00America/Boa_Vista
-04:00America/Campo_Grande
-04:00America/Caracas
-04:00America/Cuiaba
-04:00America/Curacao
-04:00America/Dominica
-04:00America/Glace_Bay
-04:00America/Goose_Bay
-04:00America/Grenada
-04:00America/Guadeloupe
-04:00America/Guyana
-04:00America/HalifaxCanada/Atlantic
-04:00America/Kralendijk
-04:00America/La_Paz
-04:00America/Lower_Princes
-04:00America/ManausBrazil/West
-04:00America/Marigot
-04:00America/Martinique
-04:00America/Moncton
-04:00America/Montserrat
-04:00America/Port_of_SpainAmerica/Virgin
-04:00America/Porto_Velho
-04:00America/Puerto_Rico
-04:00America/SantiagoChile/Continental
-04:00America/Santo_Domingo
-04:00America/St_Barthelemy
-04:00America/St_Kitts
-04:00America/St_Lucia
-04:00America/St_Thomas
-04:00America/St_Vincent
-04:00America/Thule
-04:00America/Tortola
-04:00Atlantic/Bermuda
-04:00Etc/GMT+4
-03:30America/St_JohnsCanada/Newfoundland
-03:00America/Araguaina
-03:00America/Argentina/Buenos_AiresAmerica/Buenos_Aires
-03:00America/Argentina/CatamarcaAmerica/Argentina/ComodRivadavia, America/Catamarca
-03:00America/Argentina/CordobaAmerica/Cordoba, America/Rosario
-03:00America/Argentina/JujuyAmerica/Jujuy
-03:00America/Argentina/La_Rioja
-03:00America/Argentina/MendozaAmerica/Mendoza
-03:00America/Argentina/Rio_Gallegos
-03:00America/Argentina/Salta
-03:00America/Argentina/San_Juan
-03:00America/Argentina/San_Luis
-03:00America/Argentina/Tucuman
-03:00America/Argentina/Ushuaia
-03:00America/Bahia
-03:00America/Belem
-03:00America/Cayenne
-03:00America/Fortaleza
-03:00America/Godthab
-03:00America/Maceio
-03:00America/Miquelon
-03:00America/Montevideo
-03:00America/Paramaribo
-03:00America/Punta_Arenas
-03:00America/Recife
-03:00America/Santarem
-03:00America/Sao_PauloBrazil/East
-03:00Antarctica/Palmer
-03:00Antarctica/Rothera
-03:00Atlantic/Stanley
-03:00Etc/GMT+3
-02:00America/NoronhaBrazil/DeNoronha
-02:00Atlantic/South_Georgia
-02:00Etc/GMT+2
-01:00America/Scoresbysund
-01:00Atlantic/Azores
-01:00Atlantic/Cape_Verde
-01:00Etc/GMT+1
+00:00Africa/AbidjanAfrica/Timbuktu
+00:00Africa/Accra
+00:00Africa/Bamako
+00:00Africa/Banjul
+00:00Africa/Bissau
+00:00Africa/Casablanca
+00:00Africa/Conakry
+00:00Africa/Dakar
+00:00Africa/El_Aaiun
+00:00Africa/Freetown
+00:00Africa/Lome
+00:00Africa/Monrovia
+00:00Africa/Nouakchott
+00:00Africa/Ouagadougou
+00:00America/Danmarkshavn
+00:00Antarctica/Troll
+00:00Atlantic/Canary
+00:00Atlantic/FaroeAtlantic/Faeroe
+00:00Atlantic/Madeira
+00:00Atlantic/ReykjavikIceland
+00:00Atlantic/St_Helena
+00:00Etc/GMTEtc/GMT+0, Etc/GMT-0, Etc/GMT0, Etc/Greenwich, GMT, GMT+0, GMT-0, GMT0, Greenwich
+00:00Etc/UCTUCT
+00:00Etc/UTCEtc/Universal, Etc/Zulu, Universal, Zulu
+00:00Europe/DublinEire
+00:00Europe/Guernsey
+00:00Europe/Isle_of_Man
+00:00Europe/Jersey
+00:00Europe/LisbonPortugal
+00:00Europe/LondonEurope/Belfast, GB, GB-Eire
+00:00UTC
+00:00WET
+01:00Africa/Algiers
+01:00Africa/Bangui
+01:00Africa/Brazzaville
+01:00Africa/Ceuta
+01:00Africa/Douala
+01:00Africa/Kinshasa
+01:00Africa/Lagos
+01:00Africa/Libreville
+01:00Africa/Luanda
+01:00Africa/Malabo
+01:00Africa/Ndjamena
+01:00Africa/Niamey
+01:00Africa/Porto-Novo
+01:00Africa/Sao_Tome
+01:00Africa/Tunis
+02:00Africa/Windhoek
+01:00Arctic/Longyearbyen
+01:00CET
+01:00Etc/GMT-1
+01:00Europe/Amsterdam
+01:00Europe/Andorra
+01:00Europe/Belgrade
+01:00Europe/Berlin
+01:00Europe/Bratislava
+01:00Europe/Brussels
+01:00Europe/Budapest
+01:00Europe/Busingen
+01:00Europe/Copenhagen
+01:00Europe/Gibraltar
+01:00Europe/Ljubljana
+01:00Europe/Luxembourg
+01:00Europe/Madrid
+01:00Europe/Malta
+01:00Europe/Monaco
+01:00Europe/OsloAtlantic/Jan_Mayen
+01:00Europe/Paris
+01:00Europe/Podgorica
+01:00Europe/Prague
+01:00Europe/Rome
+01:00Europe/San_Marino
+01:00Europe/Sarajevo
+01:00Europe/Skopje
+01:00Europe/Stockholm
+01:00Europe/Tirane
+01:00Europe/Vaduz
+01:00Europe/Vatican
+01:00Europe/Vienna
+01:00Europe/WarsawPoland
+01:00Europe/Zagreb
+01:00Europe/Zurich
+01:00MET
+02:00Africa/Blantyre
+02:00Africa/Bujumbura
+02:00Africa/CairoEgypt
+02:00Africa/Gaborone
+02:00Africa/Harare
+02:00Africa/Johannesburg
+02:00Africa/Khartoum
+02:00Africa/Kigali
+02:00Africa/Lubumbashi
+02:00Africa/Lusaka
+02:00Africa/Maputo
+02:00Africa/Maseru
+02:00Africa/Mbabane
+02:00Africa/TripoliLibya
+02:00Asia/Amman
+02:00Asia/Beirut
+02:00Asia/Damascus
+02:00Asia/Famagusta
+02:00Asia/Gaza
+02:00Asia/Hebron
+02:00Asia/JerusalemAsia/Tel_Aviv, Israel
+02:00Asia/Nicosia
+02:00EET
+02:00Etc/GMT-2
+02:00Europe/Athens
+02:00Europe/Bucharest
+02:00Europe/ChisinauEurope/Tiraspol
+02:00Europe/Helsinki
+02:00Europe/Kaliningrad
+02:00Europe/Kiev
+02:00Europe/Mariehamn
+02:00Europe/Nicosia
+02:00Europe/Riga
+02:00Europe/Sofia
+02:00Europe/Tallinn
+02:00Europe/Uzhgorod
+02:00Europe/Vilnius
+02:00Europe/Zaporozhye
+03:00Africa/Addis_Ababa
+03:00Africa/Asmara
+03:00Africa/Dar_es_Salaam
+03:00Africa/Djibouti
+03:00Africa/Juba
+03:00Africa/Kampala
+03:00Africa/Mogadishu
+03:00Africa/NairobiAfrica/Asmera
+03:00Antarctica/Syowa
+03:00Asia/Aden
+03:00Asia/Baghdad
+03:00Asia/Bahrain
+03:00Asia/Istanbul
+03:00Asia/Kuwait
+03:00Asia/Qatar
+03:00Asia/Riyadh
+03:00Etc/GMT-3
+03:00Europe/IstanbulTurkey
+03:00Europe/Kirov
+03:00Europe/Minsk
+03:00Europe/MoscowW-SU
+03:00Europe/Simferopol
+03:00Indian/Antananarivo
+03:00Indian/Comoro
+03:00Indian/Mayotte
+03:30Asia/TehranIran
+04:00Asia/Baku
+04:00Asia/Dubai
+04:00Asia/Muscat
+04:00Asia/Tbilisi
+04:00Asia/Yerevan
+04:00Etc/GMT-4
+04:00Europe/Astrakhan
+04:00Europe/Samara
+04:00Europe/Saratov
+04:00Europe/Ulyanovsk
+04:00Europe/Volgograd
+04:00Indian/Mahe
+04:00Indian/Mauritius
+04:00Indian/Reunion
+04:30Asia/Kabul
+05:00Antarctica/Mawson
+05:00Asia/Aqtau
+05:00Asia/Aqtobe
+05:00Asia/AshgabatAsia/Ashkhabad
+05:00Asia/Atyrau
+05:00Asia/Dushanbe
+05:00Asia/Karachi
+05:00Asia/Oral
+05:00Asia/Samarkand
+05:00Asia/Tashkent
+05:00Asia/Yekaterinburg
+05:00Etc/GMT-5
+05:00Indian/Kerguelen
+05:00Indian/Maldives
+05:30Asia/Colombo
+05:30Asia/KolkataAsia/Calcutta
+05:45Asia/KathmanduAsia/Katmandu
+06:00Antarctica/Vostok
+06:00Asia/Almaty
+06:00Asia/Bishkek
+06:00Asia/DhakaAsia/Dacca
+06:00Asia/Omsk
+06:00Asia/Qyzylorda
+06:00Asia/ThimphuAsia/Thimbu
+06:00Asia/UrumqiAsia/Kashgar
+06:00Etc/GMT-6
+06:00Indian/Chagos
+06:30Asia/YangonAsia/Rangoon
+06:30Indian/Cocos
+07:00Antarctica/Davis
+07:00Asia/Bangkok
+07:00Asia/Barnaul
+07:00Asia/Ho_Chi_MinhAsia/Saigon
+07:00Asia/Hovd
+07:00Asia/Jakarta
+07:00Asia/Krasnoyarsk
+07:00Asia/Novokuznetsk
+07:00Asia/Novosibirsk
+07:00Asia/Phnom_Penh
+07:00Asia/Pontianak
+07:00Asia/Tomsk
+07:00Asia/Vientiane
+07:00Etc/GMT-7
+07:00Indian/Christmas
+08:00Antarctica/Casey
+08:00Asia/Brunei
+08:00Asia/Choibalsan
+08:00Asia/Hong_KongHongkong
+08:00Asia/Irkutsk
+08:00Asia/Kuala_Lumpur
+08:00Asia/Kuching
+08:00Asia/MacauAsia/Macao
+08:00Asia/MakassarAsia/Ujung_Pandang
+08:00Asia/Manila
+08:00Asia/ShanghaiAsia/Chongqing, Asia/Chungking, Asia/Harbin, PRC
+08:00Asia/SingaporeSingapore
+08:00Asia/TaipeiROC
+08:00Asia/UlaanbaatarAsia/Ulan_Bator
+08:00Australia/PerthAustralia/West
+08:00Etc/GMT-8
+08:45Australia/Eucla
+09:00Asia/Chita
+09:00Asia/Dili
+09:00Asia/Jayapura
+09:00Asia/Khandyga
+09:00Asia/Pyongyang
+09:00Asia/SeoulROK
+09:00Asia/TokyoJapan
+09:00Asia/Yakutsk
+09:00Etc/GMT-9
+09:00Pacific/Palau
+09:30Australia/AdelaideAustralia/South
+09:30Australia/Broken_HillAustralia/Yancowinna
+09:30Australia/DarwinAustralia/North
+10:00Antarctica/DumontDUrville
+10:00Asia/Ust-Nera
+10:00Asia/Vladivostok
+10:00Australia/BrisbaneAustralia/Queensland
+10:00Australia/Currie
+10:00Australia/HobartAustralia/Tasmania
+10:00Australia/Lindeman
+10:00Australia/MelbourneAustralia/Victoria
+10:00Australia/SydneyAustralia/ACT, Australia/Canberra, Australia/NSW
+10:00Etc/GMT-10
+10:00Pacific/ChuukPacific/Truk, Pacific/Yap
+10:00Pacific/Guam
+10:00Pacific/Port_Moresby
+10:00Pacific/Saipan
+10:30Australia/Lord_HoweAustralia/LHI
+11:00Antarctica/Macquarie
+11:00Asia/Magadan
+11:00Asia/Sakhalin
+11:00Asia/Srednekolymsk
+11:00Etc/GMT-11
+11:00Pacific/Bougainville
+11:00Pacific/Efate
+11:00Pacific/Guadalcanal
+11:00Pacific/Kosrae
+11:00Pacific/Norfolk
+11:00Pacific/Noumea
+11:00Pacific/PohnpeiPacific/Ponape
+12:00Antarctica/McMurdo
+12:00Asia/Anadyr
+12:00Asia/Kamchatka
+12:00Etc/GMT-12
+12:00Pacific/AucklandAntarctica/South_Pole, NZ
+12:00Pacific/Fiji
+12:00Pacific/Funafuti
+12:00Pacific/KwajaleinKwajalein
+12:00Pacific/Majuro
+12:00Pacific/Nauru
+12:00Pacific/Tarawa
+12:00Pacific/Wake
+12:00Pacific/Wallis
+12:45Pacific/ChathamNZ-CHAT
+13:00Etc/GMT-13
+13:00Pacific/Apia
+13:00Pacific/Enderbury
+13:00Pacific/Fakaofo
+13:00Pacific/Tongatapu
+14:00Etc/GMT-14
+14:00Pacific/Kiritimati

参考

  1. java SimpleDateFormat 设置时区格式化时间
  2. 国际时区 TimeZone ID 列表
  3. oda-Time - Java data and time API - Time Zones