Comprehensive validation 🔎, 30+ fixes integrated/added 🔨🐛, optimized performance 🚀

[klondikedragon]

This package is amazing and hugely popular, and has been the best package for automatic date parsing in go for years! ⭐

Thanks @araddon for crafting this package with love over the years!!

I've been using this while developing a new cloud-based log aggregation/search/visualization product, and I've found that there are three major opportunities for improvement for my particular use case:

• The package does not strictly validate its input, leading to many false positives. This is OK if you know for sure the input matches one of the known formats, but cannot be trusted if the input could be anything and you only want a returned date/time if it definitely matches a known format.
• While still being far more efficient than the "shotgun" parsing approach, the package currently allocates a relatively large amount of memory (several times the average input size), which can add up when parsing megabytes of date strings per second in a high-throughput microservice. It can also be relatively slow when parsing a string that doesn't match a known format and can allocate even more memory in this case, due to custom error messages that include contextual details.
• There are a lot of unmerged one-off contributions that haven't been merged and need to be made coherent with each other.

This PR addresses all 3 opportunities:

• Validation: it now comprehensively validates the input in the following ways: at each point in the state machine, it makes sure there are cases for all possibilities, and any invalid possibility will fail; additionally, it makes sure that the entire format string was specifically set (excluding any trailing punctuation, which can be safely ignored). False positives should be extremely rare now (hard to prove they don't exist).
• Memory Efficiency: bytes allocated have been reduced by 90%, and parsing some formats are zero allocation. The SimpleErrorMessages option was added (off by default) that greatly speeds up the case where a string does not match a known format -- with the option on, this case is now 4x faster and produces almost no allocations.
• Merge & Integrate Community Fixes: Fixes for nearly all of the pending issues from the community (including open pull requests) have been incorporated and adapted.

In the process of going through the state machine comprehensively for validation, redundant code/states were merged, and support was added for certain edge cases (for example, some date formats did not support being followed by times).

The example and README.md were updated to incorporate all of the newly supported formats and edge cases. More details on how to properly interpret returned location information with respect to abbreviated timezones was added.

BREAKING -- the package now requires go >= 1.20 to support memory optimizations converting from []byte to string in key places.

A huge thanks to all who posted issues and contributed PRs -- while the PRs were unable to be merged directly because the validation changes were so major, the ideas of all these contributions and the associated test cases were incorporated. Here's credit for all of the issues fixes and contributions in this PR as well as a summary of additional fixes added:

• This PR started directly building from Supported extra whitespaces where I needed them + many other changes #151 by @arran4 that fixes Cannot parse 2015-06-10 00:00:00 GMT+02:00 #78 (support for GMT+offset, support additional whitespace in certain places, and other changes)
• Fix [BUG] some coma-separated words are considered as a time #145 - words falsely detected as a valid format - fixed by comprehensive validation
• Fix "1.jpg" is incorrectly matched as a date #108 - "1.jpg" incorrectly parsed as valid - fixed by comprehensive validation
• Fix Names and addresses are parsed without an error #98 - names and addresses incorrectly parsed as valid - fixed by comprehensive validation
• Fix Issues with parsing 12:xx am #150 - fix parsing "am" as "pm"
• Fix can't parse Thu Jan 28 2021 15:28:21 GMT+0000 (Coordinated Universal Time) #157 - fix parsing of Thu Jan 28 2021 15:28:21 GMT+0000 format
• Fix Support ":" as separator for fractional seconds in the absence of a zone offset #137 - variant of ':' as separator for fraction seconds - adapted support ":" as separator for fractional seconds #138 by @dferstay
• Fix Support eurodate format (dd-mmm-yy with a digit month) #139 - add support for dd-mm-yyyy (digit month) - adapt Add support for dd-mm-yyyy (digit month) formats #140 by @dferstay (also fixes Cannot parse "13-02-2015", but successfully parses "13/02/2015" #155)
• Fix Support "yyyy mmm dd" dates where mmm is an alpha literal #141 - add support for yyyy mon dd format - adapt Add support for dates of the form "yyyy mmm dd" where mmm is alpha #142 by @dferstay
• Fix Support combined datetime format with subseconds (yyyyMMddhhmmss.SSS)  #143 - add support for yyyymmddhhmmss.SSS format - adapt support combined datetime format with subseconds (yyyyMMddhhmmss.SSS) #144 by @dferstay
• Fix Timezone 'Z' not parsed for time string with milliseconds after period #130 - fix timezone parsing after fractional seconds - adapted issue-130 fix timezone detection issue after timePeriod #131 by @zifengyu
• Fix Not finding time for CET with millisecond and time with nanosecond #123 - fix parsing of fractional seconds for certain formats (e.g., 2017-04-03 22:32:14.322 CET)
• Fix Date format not being parsed #109 - fix parsing of Sun, 07 Jun 2020 00:00:00 +0100 format
• Fix bug in comment Cant parse a simple date #100 (comment) - fix parsing of 1 April 2022 23:59 format (time after certain date formats)
• Fix Date format of DD.MM.YYYY HH:MM:SS not supported #129 - fix ambiguous and PreferMonthFirst parsing for format mm.dd.yyyy (time) - adapted fix problem with date format 02.01.2006 #133 by @mehanizm (also fixes Better support for dot-format (e.g. 13.1.2009) with ambiguousMMDD #91 and Request: Support for DD/MM/YYYY #28)
• Fix Some forms of PM indicator combined with time zone don't work #149 - support PMDT and AMT time zones and validate that AM/PM indicators only appear at most once
• Fix Suffixes are not supported in some formats #127 - add support for dd[th,nd,st,rd] Month yyyy format - adapt New date format 1st November 2020 #128 by @krhubert
• Fix Does not respect offset for format (time) UTC[+-]NNNN #158 - fix parsing for format (time) UTC[+-]NNNN
• Add support for mm/dd/yyyy, hh:mm:ss format - adapt added comma format like 04/2/2014, 03:00:37 #156 by @BrianLeishman
• Add support for yyyy.mm.dd (time) format - adapt Bug parsing 2014.02.13 00:00:00 ? #134 by @jmdacruz, and add cases expected to fail to TestParseErrors unit test
• Add support for git log format (e.g., Thu Apr 7 15:13:13 2005 -0700) - adapt commit 99d9682 from Add support for git date format #92 by @jiangxin (merge timeWsYearOffset case and add validation)
• Add support for RabbitMQ log format (e.g., dd-mon-yyyy::hh:mm:ss) - adapt rabbitmq log datetime support #122 by @bizy01
• Expand support for Chinese date formats, and allow times to follow - adapt expand Chinese time format #132 by @xwjdsh
• Add support for mon/dd/yyyy format, e.g., Oct/31/1970
• Add support for dd-month-year format
• Extend format yyyy-mon-dd to allow times to follow it. Also allow full month name instead of just abbreviated.
• Fix the case for ambiguous date/time in the mm:dd:yyyy format
• Allow full day name before month (e.g., Monday January 4th, 2017)
• Additional fixes for mm.dd.yyyy (time) format
• Fix ambiguous parsing for mm/dd formats that start with a weekday

Also adds tests to verify that the following stay fixed:

• Cannot parse date: 190910 11:51:49 #94 - mysql log format such as 190910 11:51:49
• Date Parse Fails on dates returned by Workfront in the European timezone #117 - fractional seconds after ':'

[elliot40404]

great work @klondikedragon . How can i start using this?

[klondikedragon]

I'll go ahead and fork this package. I'm renaming the main branch as part of that.

[klondikedragon]

The fork is complete and published as v0.1.0 -- again, a huge thanks to @araddon for authoring and maintaining this package for so many years!

The fork is available using go get github.com/itlightning/dateparse -- issues and PRs are welcome.

@elliot40404 @arran4 @jmdacruz -- see what you think and how this updated package works! If this looks good and after incorporating feedback, I think I'll publish a v1.0.0 at some point soon. I'm also curious to get feedback on my log management project too, check out the site/discord if you're interested. Thanks!

[arran4]

What happened?

[klondikedragon]

@arran4 sorry... I was cleaning up old branches when releasing a new version of https://github.com/itlightning/dateparse (part of work to add automated config-free syslog parsing/ingestion to https://sparklogs.com/ ), and I forgot that deleting the old master branch would close this PR (I'm using main branch actively now). I've recreated the branch so this PR can remain open 🙂

v0.2.1 of the github.com/itlightning/dateparse package is now available with support for additional syslog RFC3164/RFC5424 time formats (and their many variants). Dependencies were also updated. Use/development of the package continues to be active there and feedback welcome.